[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:16 dist-test-slave-6042 krb5kdc[8923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:16 dist-test-slave-6042 krb5kdc[8923](info): set up 2 sockets
Sep 25 11:06:16 dist-test-slave-6042 krb5kdc[8923](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 11:06:17.455092 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:17 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:17.643662 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37665
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:17.980310 8936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:17.980679 8936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:17.980840 8936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:17.988250 8936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:17.988418 8936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:17.988528 8936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:17.988636 8936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:17.988713 8936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:17.999130 8936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:37665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:18.000660 8936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:18.002019 8936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:18.014917 8941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:18.014998 8942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:18.014998 8944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:18.015307 8936 server_base.cc:1034] running on GCE node
I20260925 11:06:18.016443 8936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:18.018191 8936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:18.019411 8936 hybrid_clock.cc:648] HybridClock initialized: now 1790334378019346 us; error 48 us; skew 500 ppm
I20260925 11:06:18.019798 8936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:18 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:18.028609 8936 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:18.030354 8936 webserver.cc:460] Webserver started at http://127.8.181.62:37723/ using document root <none> and password file <none>
I20260925 11:06:18.031126 8936 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:18.031284 8936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:18.031610 8936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:18.034811 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "50ff1df6a57a4919a692b58eed64bda0"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7725b6ec2b08e8b725c0538ec56ac1ec"
server_key_iv: "ef9009a646264e209a694cadb30583f6"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.035607 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "50ff1df6a57a4919a692b58eed64bda0"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7725b6ec2b08e8b725c0538ec56ac1ec"
server_key_iv: "ef9009a646264e209a694cadb30583f6"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.041062 8936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:18.044961 8951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:18.046422 8936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:18.046694 8936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "50ff1df6a57a4919a692b58eed64bda0"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7725b6ec2b08e8b725c0538ec56ac1ec"
server_key_iv: "ef9009a646264e209a694cadb30583f6"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.046913 8936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:18.073283 8936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:18.078188 8936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:18.078521 8936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:18.098851 8936 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:37665
I20260925 11:06:18.098893 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:37665 every 8 connection(s)
I20260925 11:06:18.100253 8936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:18.102285 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 8936
I20260925 11:06:18.102643 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:18.104264 9003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:18.104561 8916 external_mini_cluster.cc:1428] Setting key 5d0f9cc60122c29d0fea79a4ef40ebc6
I20260925 11:06:18.115890 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0: Bootstrap starting.
I20260925 11:06:18.119118 9003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:18.120175 9003 log.cc:824] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:18.122951 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0: No bootstrap required, opened a new log
I20260925 11:06:18.131834 9003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } }
I20260925 11:06:18.132280 9003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:18.132413 9003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ff1df6a57a4919a692b58eed64bda0, State: Initialized, Role: FOLLOWER
Sep 25 11:06:18 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:18.133319 9003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } }
I20260925 11:06:18.133618 9003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:18.133777 9003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:18.133952 9003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:18.137454 9003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } }
I20260925 11:06:18.138156 9003 leader_election.cc:302] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ff1df6a57a4919a692b58eed64bda0; no voters:
I20260925 11:06:18.138926 9003 leader_election.cc:288] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:18.139120 9008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:18.140652 9008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [term 1 LEADER]: Becoming Leader. State: Replica: 50ff1df6a57a4919a692b58eed64bda0, State: Running, Role: LEADER
I20260925 11:06:18.141377 9003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:18.141319 9008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } }
I20260925 11:06:18.149053 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:18.108479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.8.181.62:37665)
0925 11:06:18.109385 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:18.109406 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:18.110488 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 11:06:18.119454 (+ 8966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:18.119476 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:18.119861 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:18.120249 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:18.123184 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.125217 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.127038 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.128102 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.128292 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:18.134979 (+ 6687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:18.134991 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:18.135012 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:18.135069 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:18.139459 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:18.140690 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:18.140703 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:18.140711 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:18.140938 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:18.141607 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:18.141612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:18.141615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:18.142364 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:18.145800 (+ 3436us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:18.146197 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 11:06:18.146857 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":404,"threads_started":1}
I20260925 11:06:18.151199 9010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ff1df6a57a4919a692b58eed64bda0. Latest consensus state: current_term: 1 leader_uuid: "50ff1df6a57a4919a692b58eed64bda0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } } }
I20260925 11:06:18.151638 9010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:18.152441 9009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ff1df6a57a4919a692b58eed64bda0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ff1df6a57a4919a692b58eed64bda0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37665 } } }
I20260925 11:06:18.152776 9009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:18.157168 9022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:18.162523 9022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:18.172179 9022 catalog_manager.cc:1347] Generated new cluster ID: de8758164ce841398420a7ca0e46e53c
I20260925 11:06:18.172304 9022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:18.182452 9022 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:18.183813 9022 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:18.190078 9022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ff1df6a57a4919a692b58eed64bda0: Generated new TSK 0
I20260925 11:06:18.190997 9022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:18.247488 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:18.620723 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:18.621047 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:18.621238 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:18.629745 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:18.629876 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:18.630062 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:18.640808 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:18.642365 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:18.643904 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:18.658319 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:18.658340 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:18.658371 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:18.659205 9029 server_base.cc:1034] running on GCE node
I20260925 11:06:18.659924 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:18.660914 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:18.662159 9029 hybrid_clock.cc:648] HybridClock initialized: now 1790334378662111 us; error 61 us; skew 500 ppm
I20260925 11:06:18.662547 9029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:18 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:18.666046 9029 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:18.667712 9029 webserver.cc:460] Webserver started at http://127.8.181.1:45819/ using document root <none> and password file <none>
I20260925 11:06:18.668430 9029 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:18.668581 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:18.668896 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:18.671959 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "17af862119ef4451a296e54e575a9f04"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7b114859fcb1e89ce0737700bb8611a0"
server_key_iv: "6102e63a455f5d5b6bca5185d7a59fd4"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.672749 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "17af862119ef4451a296e54e575a9f04"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7b114859fcb1e89ce0737700bb8611a0"
server_key_iv: "6102e63a455f5d5b6bca5185d7a59fd4"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.678436 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 11:06:18.682745 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:18.684199 9029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:18.684405 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "17af862119ef4451a296e54e575a9f04"
format_stamp: "Formatted at 2026-09-25 11:06:18 on dist-test-slave-6042"
server_key: "7b114859fcb1e89ce0737700bb8611a0"
server_key_iv: "6102e63a455f5d5b6bca5185d7a59fd4"
server_key_version: "encryptionkey@0"
I20260925 11:06:18.684712 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:18.712278 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:18.717069 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:18.717427 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:18.718571 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:18.719991 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:18.720069 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:18.720160 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:18.720208 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:18.756992 9029 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:36533
I20260925 11:06:18.757037 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:36533 every 8 connection(s)
I20260925 11:06:18.758335 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:18.758796 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9029
I20260925 11:06:18.759147 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:18.760066 8916 external_mini_cluster.cc:1428] Setting key 513b6273d69bc2b6ca595d2a91ac3b8a
Sep 25 11:06:18 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334378, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:06:18.784595 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:18.761422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:53561 (local address 127.8.181.62:37665)
0925 11:06:18.762047 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:18.762056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:18.763274 (+ 1218us) server_negotiation.cc:407] Connection header received
0925 11:06:18.766618 (+ 3344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:18.766623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:18.766695 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:18.766792 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:18.768918 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.770211 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.772184 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.772927 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.773092 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:18.777233 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:18.777241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:18.777246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:18.777290 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:18.780795 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:18.781654 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:18.781662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:18.781666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:18.781873 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:18.782531 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:18.782537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:18.782541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:18.782876 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:18.783041 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:18.783979 (+ 938us) server_negotiation.cc:299] Negotiation successful
0925 11:06:18.784209 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":260,"threads_started":1}
I20260925 11:06:18.785974 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:18.762024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37665 (local address 127.8.181.1:53561)
0925 11:06:18.763074 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:18.763131 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:18.766193 (+ 3062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:18.767094 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:18.767110 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:18.768029 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:18.768709 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:18.768730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.770600 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.770608 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:18.772006 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:18.772023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:18.773174 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:18.773179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:18.773397 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:18.774155 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:18.774179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:18.776970 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:18.780955 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:18.780964 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:18.780985 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:18.781512 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:18.782030 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:18.782038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:18.782045 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:18.782303 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:18.783020 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:18.783028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:18.783511 (+ 483us) client_negotiation.cc:769] Sending connection context
0925 11:06:18.783853 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 11:06:18.784408 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":482,"threads_started":1}
I20260925 11:06:18.787827 9157 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37665
I20260925 11:06:18.788118 9157 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:18.788858 9157 heartbeater.cc:511] Master 127.8.181.62:37665 requested a full tablet report, sending...
I20260925 11:06:18.791071 8968 ts_manager.cc:194] Registered new tserver with Master: 17af862119ef4451a296e54e575a9f04 (127.8.181.1:36533)
I20260925 11:06:18.792546 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:53561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:18.805971 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:19.131942 9163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:19.132210 9163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:19.132369 9163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:19.139726 9163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:19.139838 9163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:19.140038 9163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:19.150182 9163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:19.151604 9163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:19.153084 9163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:19.166815 9168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:19.166808 9169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:19.167474 9171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:19.167905 9163 server_base.cc:1034] running on GCE node
I20260925 11:06:19.168498 9163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:19.169468 9163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:19.170704 9163 hybrid_clock.cc:648] HybridClock initialized: now 1790334379170644 us; error 55 us; skew 500 ppm
I20260925 11:06:19.171012 9163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:19.174108 9163 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:19.175572 9163 webserver.cc:460] Webserver started at http://127.8.181.2:41509/ using document root <none> and password file <none>
I20260925 11:06:19.176249 9163 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:19.176347 9163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:19.176597 9163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:19.179481 9163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "434ef5335c89435b8964a9da4e3d0bba"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "adc7f03973547f5e22c06fef268a807f"
server_key_iv: "343409880cc0a2ae6836c44afcfd671d"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.180181 9163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "434ef5335c89435b8964a9da4e3d0bba"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "adc7f03973547f5e22c06fef268a807f"
server_key_iv: "343409880cc0a2ae6836c44afcfd671d"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.185266 9163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:19.188666 9178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.190079 9163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:19.190256 9163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "434ef5335c89435b8964a9da4e3d0bba"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "adc7f03973547f5e22c06fef268a807f"
server_key_iv: "343409880cc0a2ae6836c44afcfd671d"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.190529 9163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:19.206849 9163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:19.211238 9163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:19.211478 9163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:19.212404 9163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:19.213701 9163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:19.213773 9163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.213876 9163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:19.213922 9163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.247260 9163 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:38849
I20260925 11:06:19.247325 9290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:38849 every 8 connection(s)
I20260925 11:06:19.248730 9163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:19.253314 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9163
I20260925 11:06:19.253567 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:19.254026 8916 external_mini_cluster.cc:1428] Setting key 87edda13597e557408ea45c50ca0aa55
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:19.269380 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.251470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:39911 (local address 127.8.181.62:37665)
0925 11:06:19.251674 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.251679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.253082 (+ 1403us) server_negotiation.cc:407] Connection header received
0925 11:06:19.255229 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.255235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.255327 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.255441 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.257080 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.257791 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.259409 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.259854 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.259992 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.263502 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:19.263507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:19.263511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:19.263542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:19.266266 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.267045 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.267057 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.267060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.267191 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.267632 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.267638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.267646 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.267925 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:19.268094 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:19.268776 (+ 682us) server_negotiation.cc:299] Negotiation successful
0925 11:06:19.269015 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:19.270354 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.252066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37665 (local address 127.8.181.2:39911)
0925 11:06:19.252864 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:19.252922 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:19.254957 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:19.255679 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:19.255688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:19.256340 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:19.256904 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:19.256920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.257953 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.257963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:19.259262 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:19.259274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.260082 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.260089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:19.260323 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.261101 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:19.261129 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:19.263268 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:19.266408 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:19.266417 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:19.266436 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:19.266909 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:19.267306 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:19.267312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:19.267316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:19.267510 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:19.268055 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:19.268063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:19.268411 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 11:06:19.268682 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 11:06:19.269206 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":302,"threads_started":1}
I20260925 11:06:19.271936 9291 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37665
I20260925 11:06:19.272253 9291 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:06:19.273164 9291 heartbeater.cc:511] Master 127.8.181.62:37665 requested a full tablet report, sending...
I20260925 11:06:19.274891 8968 ts_manager.cc:194] Registered new tserver with Master: 434ef5335c89435b8964a9da4e3d0bba (127.8.181.2:38849)
I20260925 11:06:19.276139 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:39911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:19.294176 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:19.620561 9296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:19.620846 9296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:19.621001 9296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:19.628492 9296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:19.628608 9296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:19.628809 9296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:19.638622 9296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37665
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:19.639962 9296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:19.641265 9296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:19.654721 9301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:19.654651 9302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:19.654699 9304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:19.655614 9296 server_base.cc:1034] running on GCE node
I20260925 11:06:19.656253 9296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:19.657269 9296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:19.658565 9296 hybrid_clock.cc:648] HybridClock initialized: now 1790334379658478 us; error 74 us; skew 500 ppm
I20260925 11:06:19.658922 9296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:19.662204 9296 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:19.663712 9296 webserver.cc:460] Webserver started at http://127.8.181.3:39625/ using document root <none> and password file <none>
I20260925 11:06:19.664412 9296 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:19.664516 9296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:19.664755 9296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:19.667735 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "f6adabdb4e804a4cb7a96d7cae748dd6"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "6f6127a71b19db94029fb2027017ebe3"
server_key_iv: "27a992333da81593917b2dc9fb52524b"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.668484 9296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "f6adabdb4e804a4cb7a96d7cae748dd6"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "6f6127a71b19db94029fb2027017ebe3"
server_key_iv: "27a992333da81593917b2dc9fb52524b"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.673977 9296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 11:06:19.677436 9311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.678985 9296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:19.679163 9296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "f6adabdb4e804a4cb7a96d7cae748dd6"
format_stamp: "Formatted at 2026-09-25 11:06:19 on dist-test-slave-6042"
server_key: "6f6127a71b19db94029fb2027017ebe3"
server_key_iv: "27a992333da81593917b2dc9fb52524b"
server_key_version: "encryptionkey@0"
I20260925 11:06:19.679373 9296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:19.700330 9296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:19.705076 9296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:19.705425 9296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:19.706430 9296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:19.707850 9296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:19.707930 9296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.708087 9296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:19.708140 9296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:19.742370 9296 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:34949
I20260925 11:06:19.742410 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:34949 every 8 connection(s)
I20260925 11:06:19.743744 9296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:19.752116 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9296
I20260925 11:06:19.752319 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:19.752727 8916 external_mini_cluster.cc:1428] Setting key 454b0d8d3133f1be28b598285a3dc1c9
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:19.764438 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.746335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:58321 (local address 127.8.181.62:37665)
0925 11:06:19.746607 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.746612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.747780 (+ 1168us) server_negotiation.cc:407] Connection header received
0925 11:06:19.749677 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.749683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.749800 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.749912 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.751630 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.752510 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.754367 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.755036 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.755183 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.758872 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:19.758878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:19.758883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:19.758921 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:19.761467 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.762149 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.762155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.762158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.762281 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.762762 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.762767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.762771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.763053 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:19.763224 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:19.763865 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 11:06:19.764089 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:06:19.765309 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.746838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37665 (local address 127.8.181.3:58321)
0925 11:06:19.747557 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:19.747661 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:19.749396 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:19.750181 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:19.750189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:19.750846 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:19.751464 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:19.751481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.752663 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.752667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:19.754223 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:19.754240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.755303 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.755310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:19.755605 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.756411 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:19.756447 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:19.758671 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:19.761614 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:19.761622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:19.761636 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:19.762021 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:19.762388 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:19.762394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:19.762397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:19.762648 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:19.763182 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:19.763189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:19.763506 (+ 317us) client_negotiation.cc:769] Sending connection context
0925 11:06:19.763757 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 11:06:19.764245 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260925 11:06:19.766776 9424 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37665
I20260925 11:06:19.767104 9424 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:19.767868 9424 heartbeater.cc:511] Master 127.8.181.62:37665 requested a full tablet report, sending...
I20260925 11:06:19.769563 8967 ts_manager.cc:194] Registered new tserver with Master: f6adabdb4e804a4cb7a96d7cae748dd6 (127.8.181.3:34949)
I20260925 11:06:19.770458 8967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:58321
I20260925 11:06:19.781832 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:06:19.797254 9157 heartbeater.cc:503] Master 127.8.181.62:37665 was elected leader, sending a full tablet report...
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:19.828478 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.812522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.8.181.1:36533)
0925 11:06:19.813256 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.813263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.813291 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:06:19.813353 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.813359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.813583 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.813712 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.815278 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.816386 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.817676 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.818266 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.818378 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.822443 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:19.822456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:19.822584 (+ 128us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:19.822649 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:19.825688 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.826383 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.826390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.826392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.826690 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.827063 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.827067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.827069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.827408 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:19.827699 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:19.827863 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 11:06:19.828070 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":378,"threads_started":1}
W20260925 11:06:19.829909 9072 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37300
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:19.875099 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.861295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.8.181.1:36533)
0925 11:06:19.861595 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.861600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.861621 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:06:19.861761 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.861765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.861856 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.861943 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.863509 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.864214 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.865494 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.865930 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.866036 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.869999 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:19.870004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:19.870007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:19.870041 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:19.872622 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.873256 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.873260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.873262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.873391 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.873820 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.873824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.873826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.874108 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:19.874300 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:19.874640 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 11:06:19.874807 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 11:06:19.876722 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9029
I20260925 11:06:19.890156 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.877836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.8.181.1:36533)
0925 11:06:19.878073 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.878078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.878289 (+ 211us) server_negotiation.cc:407] Connection header received
0925 11:06:19.878385 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.878388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.878444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.878590 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.879765 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.880465 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.881786 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.882191 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.882293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:19.884892 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:19.884897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:19.884900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:19.884929 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:19.887637 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.888285 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.888289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.888291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.888447 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:19.888820 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:19.888824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:19.888826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:19.889071 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:19.889266 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:19.889607 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 11:06:19.889773 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 11:06:19.890998 9152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:19.975230 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9029
I20260925 11:06:19.991950 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9163
Sep 25 11:06:19 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:20.009115 9438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:19.992923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.8.181.2:38849)
0925 11:06:19.993652 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:19.993677 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:19.993712 (+ 35us) server_negotiation.cc:407] Connection header received
0925 11:06:19.993794 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:19.993801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:19.994026 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:19.994214 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:19.995889 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.996940 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.998229 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:19.998895 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:19.999050 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:20.003320 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:20.003331 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:20.003345 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:20.003378 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:20.006165 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:20.006947 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:20.006955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:20.006957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:20.007084 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:20.007508 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:20.007514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:20.007517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:20.007857 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:20.008174 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:20.008479 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 11:06:20.008746 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":328,"threads_started":1}
I20260925 11:06:20.010571 9286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:20.094514 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9163
I20260925 11:06:20.110846 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9296
Sep 25 11:06:20 dist-test-slave-6042 krb5kdc[8923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:20.127741 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:20.111935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.8.181.3:34949)
0925 11:06:20.112702 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:20.112709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:20.112739 (+ 30us) server_negotiation.cc:407] Connection header received
0925 11:06:20.112807 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:20.112813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:20.113038 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:20.113234 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:20.114579 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:20.115606 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:20.116957 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:20.117503 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:20.117647 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:20.121952 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:20.121963 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:20.121978 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:20.122021 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:20.124811 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:20.125549 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:20.125556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:20.125559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:20.125684 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:20.126084 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:20.126091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:20.126094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:20.126427 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:20.126878 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:20.127235 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 11:06:20.127449 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":361,"threads_started":1}
I20260925 11:06:20.129175 9419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:20.211297 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9296
I20260925 11:06:20.227174 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 8936
I20260925 11:06:20.228370 8998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:20.333261 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 8936
2026-09-25T11:06:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3972 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:20 dist-test-slave-6042 krb5kdc[9445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:20 dist-test-slave-6042 krb5kdc[9445](info): set up 2 sockets
Sep 25 11:06:20 dist-test-slave-6042 krb5kdc[9445](info): commencing operation
krb5kdc: starting...
W20260925 11:06:21.430639 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:21 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:21.618585 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32829
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:21.951704 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:21.952024 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:21.952179 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:21.959769 9458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:21.959889 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:21.959992 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:21.960098 9458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:21.960181 9458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:21.970286 9458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:32829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:21.971746 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:21.973078 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:21.985651 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:21.985767 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:21.985767 9464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:21.986730 9458 server_base.cc:1034] running on GCE node
I20260925 11:06:21.987516 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:21.988946 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:21.990154 9458 hybrid_clock.cc:648] HybridClock initialized: now 1790334381990098 us; error 41 us; skew 500 ppm
I20260925 11:06:21.990646 9458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:21 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334381, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:21.994017 9458 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:21.995832 9458 webserver.cc:460] Webserver started at http://127.8.181.62:40099/ using document root <none> and password file <none>
I20260925 11:06:21.996567 9458 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:21.996673 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:21.996999 9458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:22.000203 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "dcdacd98d3344c9fbac305dd656216b8"
format_stamp: "Formatted at 2026-09-25 11:06:21 on dist-test-slave-6042"
server_key: "812ee5a43c6cbc1be7e871c18ff559c0"
server_key_iv: "6b0d8c8129ac974b241ed3db4590b065"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.000998 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "dcdacd98d3344c9fbac305dd656216b8"
format_stamp: "Formatted at 2026-09-25 11:06:21 on dist-test-slave-6042"
server_key: "812ee5a43c6cbc1be7e871c18ff559c0"
server_key_iv: "6b0d8c8129ac974b241ed3db4590b065"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.006681 9458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 11:06:22.010388 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:22.011986 9458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:22.012233 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "dcdacd98d3344c9fbac305dd656216b8"
format_stamp: "Formatted at 2026-09-25 11:06:21 on dist-test-slave-6042"
server_key: "812ee5a43c6cbc1be7e871c18ff559c0"
server_key_iv: "6b0d8c8129ac974b241ed3db4590b065"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.012470 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:22.027348 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:22.032320 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:22.032618 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:22.051726 9458 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:32829
I20260925 11:06:22.051756 9524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:32829 every 8 connection(s)
I20260925 11:06:22.053159 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:22.056813 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9458
I20260925 11:06:22.057024 9525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:22.057066 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:22.057735 8916 external_mini_cluster.cc:1428] Setting key ab04cf8e16469631cdc25beba5df73ea
I20260925 11:06:22.070919 9525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8: Bootstrap starting.
Sep 25 11:06:22 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:22.074623 9525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:22.075742 9525 log.cc:824] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:22.078684 9525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8: No bootstrap required, opened a new log
I20260925 11:06:22.081665 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:22.059678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.8.181.62:32829)
0925 11:06:22.060533 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:22.060545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:22.060585 (+ 40us) server_negotiation.cc:407] Connection header received
0925 11:06:22.063005 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:22.063028 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:22.063431 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:22.063781 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:22.065508 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.066830 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.068631 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.069303 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.069433 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:22.074203 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:22.074212 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:22.074226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:22.074267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:22.077470 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:22.078208 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:22.078214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:22.078221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:22.078380 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:22.078805 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:22.078813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:22.078817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:22.079357 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:22.079686 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:22.080204 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 11:06:22.080634 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":389,"threads_started":1}
I20260925 11:06:22.086772 9525 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } }
I20260925 11:06:22.087270 9525 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:22.087385 9525 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdacd98d3344c9fbac305dd656216b8, State: Initialized, Role: FOLLOWER
I20260925 11:06:22.088056 9525 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } }
I20260925 11:06:22.088351 9525 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:22.088479 9525 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:22.088661 9525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:22.091002 9525 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } }
I20260925 11:06:22.091531 9525 leader_election.cc:302] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcdacd98d3344c9fbac305dd656216b8; no voters:
I20260925 11:06:22.092072 9525 leader_election.cc:288] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:22.092260 9530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:22.093534 9530 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [term 1 LEADER]: Becoming Leader. State: Replica: dcdacd98d3344c9fbac305dd656216b8, State: Running, Role: LEADER
I20260925 11:06:22.094133 9530 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } }
I20260925 11:06:22.094542 9525 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:22.097584 9531 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcdacd98d3344c9fbac305dd656216b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } } }
I20260925 11:06:22.097517 9532 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdacd98d3344c9fbac305dd656216b8. Latest consensus state: current_term: 1 leader_uuid: "dcdacd98d3344c9fbac305dd656216b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdacd98d3344c9fbac305dd656216b8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32829 } } }
I20260925 11:06:22.098006 9532 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:22.098003 9531 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:22.099987 9536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:22.105835 9536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:22.115621 9536 catalog_manager.cc:1347] Generated new cluster ID: 0614d8a4921840259732025b3c9ec98b
I20260925 11:06:22.115762 9536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:22.128183 9536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:22.129550 9536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:22.145402 9536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdacd98d3344c9fbac305dd656216b8: Generated new TSK 0
I20260925 11:06:22.146327 9536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:22.203500 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:22.539196 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:22.539510 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:22.539688 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:22.547035 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:22.547191 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:22.547377 9551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:22.557367 9551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:22.558710 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:22.560031 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:22.573982 9556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:22.573922 9559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:22.573895 9557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:22.576313 9551 server_base.cc:1034] running on GCE node
I20260925 11:06:22.576936 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:22.577940 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:22.579164 9551 hybrid_clock.cc:648] HybridClock initialized: now 1790334382579124 us; error 39 us; skew 500 ppm
I20260925 11:06:22.579628 9551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:22 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:22.583312 9551 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:22.585009 9551 webserver.cc:460] Webserver started at http://127.8.181.1:44525/ using document root <none> and password file <none>
I20260925 11:06:22.585798 9551 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:22.585949 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:22.586273 9551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:22.589323 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "a9624a440cb34d43a8fea57e5acd7274"
format_stamp: "Formatted at 2026-09-25 11:06:22 on dist-test-slave-6042"
server_key: "e7833e0ab6d40edb6fa969b9282b8c08"
server_key_iv: "87235cf7568942ccc0da42464863cf72"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.590112 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "a9624a440cb34d43a8fea57e5acd7274"
format_stamp: "Formatted at 2026-09-25 11:06:22 on dist-test-slave-6042"
server_key: "e7833e0ab6d40edb6fa969b9282b8c08"
server_key_iv: "87235cf7568942ccc0da42464863cf72"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.595445 9551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 11:06:22.599491 9566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:22.600957 9551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:06:22.601146 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "a9624a440cb34d43a8fea57e5acd7274"
format_stamp: "Formatted at 2026-09-25 11:06:22 on dist-test-slave-6042"
server_key: "e7833e0ab6d40edb6fa969b9282b8c08"
server_key_iv: "87235cf7568942ccc0da42464863cf72"
server_key_version: "encryptionkey@0"
I20260925 11:06:22.601363 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:22.628747 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:22.633350 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:22.633661 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:22.634682 9551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:22.636077 9551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:22.636183 9551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:22.636329 9551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:22.636402 9551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:22.671100 9551 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33737
I20260925 11:06:22.671161 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33737 every 8 connection(s)
I20260925 11:06:22.672576 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:22.675061 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9551
I20260925 11:06:22.675287 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:22.675899 8916 external_mini_cluster.cc:1428] Setting key cda914209cfe24f1458343930201a622
Sep 25 11:06:22 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:06:22.706526 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:22.676648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:50317 (local address 127.8.181.62:32829)
0925 11:06:22.684203 (+ 7555us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:22.684212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:22.684239 (+ 27us) server_negotiation.cc:407] Connection header received
0925 11:06:22.684310 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:22.684317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:22.684416 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:22.684549 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:22.687798 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.689244 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.692756 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.693511 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.693659 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:22.698802 (+ 5143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:22.698810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:22.698817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:22.698862 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:22.702187 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:22.703399 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:22.703407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:22.703411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:22.703577 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:22.704237 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:22.704243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:22.704248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:22.704646 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:22.704798 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:22.705811 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0925 11:06:22.706079 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7389,"thread_start_us":7263,"threads_started":1}
I20260925 11:06:22.708736 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:22.676072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32829 (local address 127.8.181.1:50317)
0925 11:06:22.676857 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:22.676904 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:22.680965 (+ 4061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:22.685056 (+ 4091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:22.685123 (+ 67us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:22.686297 (+ 1174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:22.687508 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:22.687543 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.689453 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.689461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:22.692314 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:22.692335 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:22.694178 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:22.694188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:22.694548 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:22.695419 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:22.695455 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:22.698507 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:22.702423 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:22.702436 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:22.702493 (+ 57us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:22.703195 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:22.703776 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:22.703784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:22.703791 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:22.704071 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:22.704808 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:22.704819 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:22.705299 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 11:06:22.706564 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0925 11:06:22.707145 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":345,"threads_started":1}
I20260925 11:06:22.721087 9679 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32829
I20260925 11:06:22.721500 9679 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:22.722604 9679 heartbeater.cc:511] Master 127.8.181.62:32829 requested a full tablet report, sending...
I20260925 11:06:22.725544 9490 ts_manager.cc:194] Registered new tserver with Master: a9624a440cb34d43a8fea57e5acd7274 (127.8.181.1:33737)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:22.728011 9490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:50317
I20260925 11:06:22.729389 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:23.055114 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:23.055382 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:23.055550 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:23.063020 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:23.063154 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:23.063315 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:23.073179 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:23.074563 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:23.075950 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:23.089675 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:23.089588 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:23.090086 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:23.090380 9685 server_base.cc:1034] running on GCE node
I20260925 11:06:23.090999 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:23.091879 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:23.093078 9685 hybrid_clock.cc:648] HybridClock initialized: now 1790334383093040 us; error 25 us; skew 500 ppm
I20260925 11:06:23.093348 9685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:23.100937 9685 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:23.102711 9685 webserver.cc:460] Webserver started at http://127.8.181.2:37517/ using document root <none> and password file <none>
I20260925 11:06:23.103447 9685 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:23.103610 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:23.103910 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:23.107028 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "6968c3ca48c5368a604b83e52a44c453"
server_key_iv: "f4327f24a482b988bf0e78c2b0f0613f"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.107752 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "6968c3ca48c5368a604b83e52a44c453"
server_key_iv: "f4327f24a482b988bf0e78c2b0f0613f"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.112926 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:23.116514 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.118089 9685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:23.118270 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "6968c3ca48c5368a604b83e52a44c453"
server_key_iv: "f4327f24a482b988bf0e78c2b0f0613f"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.118566 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:23.138551 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:23.143060 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:23.143383 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:23.144400 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:23.145803 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:23.145876 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.146049 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:23.146099 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.180274 9685 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:39763
I20260925 11:06:23.180323 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:39763 every 8 connection(s)
I20260925 11:06:23.181555 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:23.188625 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9685
I20260925 11:06:23.188859 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:23.189285 8916 external_mini_cluster.cc:1428] Setting key 4342e9e062ef1ca04a61a9cf006eee79
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:23.204360 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.184106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:52667 (local address 127.8.181.62:32829)
0925 11:06:23.184332 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.184338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.185429 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 11:06:23.187281 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.187285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.187359 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.187465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.189056 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.189837 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.191763 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.192204 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.192344 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.196426 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.196434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.196442 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.196489 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.200400 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.201285 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.201294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.201300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.201473 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.202016 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.202024 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.202030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.202535 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.202692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.203588 (+ 896us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.203875 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:06:23.205308 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.184569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32829 (local address 127.8.181.2:52667)
0925 11:06:23.185260 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.185297 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.187000 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.187711 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.187720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.188242 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:23.188827 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.188851 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.189991 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.189997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.191610 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.191630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.192451 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.192459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.192694 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.193385 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:23.193411 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:23.196169 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:23.200605 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:23.200614 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:23.200631 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:23.201129 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:23.201644 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:23.201648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:23.201651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:23.201878 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:23.202669 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:23.202676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:23.203094 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.203425 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.204021 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
I20260925 11:06:23.207150 9813 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32829
I20260925 11:06:23.207469 9813 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:23.208269 9813 heartbeater.cc:511] Master 127.8.181.62:32829 requested a full tablet report, sending...
I20260925 11:06:23.209868 9490 ts_manager.cc:194] Registered new tserver with Master: e1c4a77fc0ab4dd19b1b3a60b7100cd8 (127.8.181.2:39763)
I20260925 11:06:23.210855 9490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:52667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:23.234964 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:23.570987 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:23.571272 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:23.571367 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:23.579066 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:23.579190 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:23.579308 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:23.589099 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:23.590332 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:23.591696 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:23.604856 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:23.604885 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:23.604921 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:23.606233 9818 server_base.cc:1034] running on GCE node
I20260925 11:06:23.606972 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:23.608017 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:23.609223 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790334383609178 us; error 33 us; skew 500 ppm
I20260925 11:06:23.609655 9818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:23.613210 9818 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:23.614833 9818 webserver.cc:460] Webserver started at http://127.8.181.3:33365/ using document root <none> and password file <none>
I20260925 11:06:23.615557 9818 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:23.615713 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:23.616070 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:23.619323 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "fc1ae858bca9446a8b96731a88fd504d"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "1e649b1c04ef47bad7ba5a447091248e"
server_key_iv: "c80b638e1b3c3a3b6b0e4f70672c923a"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.620143 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "fc1ae858bca9446a8b96731a88fd504d"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "1e649b1c04ef47bad7ba5a447091248e"
server_key_iv: "c80b638e1b3c3a3b6b0e4f70672c923a"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.625716 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:23.629112 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.630399 9818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 11:06:23.630589 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "fc1ae858bca9446a8b96731a88fd504d"
format_stamp: "Formatted at 2026-09-25 11:06:23 on dist-test-slave-6042"
server_key: "1e649b1c04ef47bad7ba5a447091248e"
server_key_iv: "c80b638e1b3c3a3b6b0e4f70672c923a"
server_key_version: "encryptionkey@0"
I20260925 11:06:23.630864 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:23.652101 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:23.656520 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:23.656853 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:23.657912 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:23.659261 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:23.659333 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.659482 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:23.659530 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:23.693933 9818 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45547
I20260925 11:06:23.694072 9945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45547 every 8 connection(s)
I20260925 11:06:23.695603 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:23.703742 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 9818
I20260925 11:06:23.703986 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:23.704483 8916 external_mini_cluster.cc:1428] Setting key 344eb1362ec56d90fd90706e5abb0ea4
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:23.716938 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.698442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52453 (local address 127.8.181.62:32829)
0925 11:06:23.698704 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.698709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.699995 (+ 1286us) server_negotiation.cc:407] Connection header received
0925 11:06:23.701780 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.701785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.701863 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.701971 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.704069 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.704917 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.707159 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.707727 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.707872 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.711378 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.711382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.711386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.711416 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.714056 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.714843 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.714847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.714849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.714967 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.715356 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.715359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.715361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.715626 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.715729 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.716510 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.716698 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 11:06:23.718109 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.699076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32829 (local address 127.8.181.3:52453)
0925 11:06:23.699827 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.699865 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.701495 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.702219 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.702228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.703017 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:23.703859 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.703891 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.705154 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.705158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.706977 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.706990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.707944 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.707950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.708183 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.708833 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:23.708860 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:23.711159 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:23.714190 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:23.714197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:23.714210 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:23.714731 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:23.715068 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:23.715071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:23.715074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:23.715253 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:23.715738 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:23.715744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:23.716161 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.716395 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.716855 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":327,"threads_started":1}
I20260925 11:06:23.719648 9946 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32829
I20260925 11:06:23.720029 9946 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:23.720880 9946 heartbeater.cc:511] Master 127.8.181.62:32829 requested a full tablet report, sending...
I20260925 11:06:23.722384 9490 ts_manager.cc:194] Registered new tserver with Master: fc1ae858bca9446a8b96731a88fd504d (127.8.181.3:45547)
I20260925 11:06:23.723243 9490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:52453
I20260925 11:06:23.732419 9679 heartbeater.cc:503] Master 127.8.181.62:32829 was elected leader, sending a full tablet report...
I20260925 11:06:23.732838 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:06:23.751566 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.739596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.8.181.62:32829)
0925 11:06:23.739768 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.739772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.740162 (+ 390us) server_negotiation.cc:407] Connection header received
0925 11:06:23.740298 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.740302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.740365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.740454 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.741868 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.742785 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.744096 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.744501 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.744606 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.746864 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.746868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.746872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.746902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.749289 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.749866 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.749870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.749872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.749997 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.750297 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.750300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.750302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.750701 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.750888 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.751171 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.751338 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 11:06:23.757978 9490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:23.765384 9490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:23.788364 9958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.777826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33737 (local address 127.0.0.1:47348)
0925 11:06:23.779126 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.779165 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.779365 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.780397 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.780404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.780446 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:23.780853 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.780872 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.782907 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.782913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.785614 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.785627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.787279 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.787285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.787475 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.787531 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.787672 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.787874 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":112,"spinlock_wait_cycles":3840,"thread_start_us":896,"threads_started":1}
I20260925 11:06:23.789110 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.777992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.8.181.1:33737)
0925 11:06:23.778722 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.778734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.779373 (+ 639us) server_negotiation.cc:407] Connection header received
0925 11:06:23.779628 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.779637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.779911 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.780122 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:23.781074 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.782744 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.785786 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.787089 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.787248 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.787480 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.788616 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.788787 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":336,"threads_started":1}
I20260925 11:06:23.789402 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.778514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.8.181.2:39763)
0925 11:06:23.779991 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.779998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.780027 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:06:23.780126 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.780132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.780337 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.780517 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:23.781435 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.782927 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.786658 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.788053 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.788176 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.788348 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.788893 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.789042 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":1814,"threads_started":1}
I20260925 11:06:23.790175 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.777920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:39763 (local address 127.0.0.1:33596)
0925 11:06:23.779163 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.779185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.779366 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.780736 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.780743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.780774 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:23.781240 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.781251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.783088 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.783108 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.786453 (+ 3345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.786488 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.788515 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.788521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.788691 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.788721 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.789460 (+ 739us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.789692 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":966,"spinlock_wait_cycles":12416,"thread_start_us":996,"threads_started":1}
I20260925 11:06:23.793864 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 2b011bbe006e4aeb81b3707296424526 (DEFAULT_TABLE table=test-table [id=171d280e9b564f62905e4b4374ceda3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:23.793862 9748 tablet_service.cc:1467] Processing CreateTablet for tablet 2b011bbe006e4aeb81b3707296424526 (DEFAULT_TABLE table=test-table [id=171d280e9b564f62905e4b4374ceda3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:23.795528 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b011bbe006e4aeb81b3707296424526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:23.796795 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b011bbe006e4aeb81b3707296424526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:23.796929 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.780049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.8.181.3:45547)
0925 11:06:23.785313 (+ 5264us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.785321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.785351 (+ 30us) server_negotiation.cc:407] Connection header received
0925 11:06:23.785434 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.785441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.785676 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.785890 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:23.786450 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.790847 (+ 4397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.793441 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.795075 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.795270 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.795561 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.796268 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.796461 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5095,"thread_start_us":4925,"threads_started":1}
I20260925 11:06:23.797343 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.779523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45547 (local address 127.0.0.1:57226)
0925 11:06:23.784665 (+ 5142us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.784684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.784788 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.785880 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.785885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.785909 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:23.786282 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.786290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.791041 (+ 4751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.791047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.793239 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.793256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.795329 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.795335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.795536 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.795570 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.795676 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.795817 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4999,"thread_start_us":4868,"threads_started":1}
I20260925 11:06:23.800875 9881 tablet_service.cc:1467] Processing CreateTablet for tablet 2b011bbe006e4aeb81b3707296424526 (DEFAULT_TABLE table=test-table [id=171d280e9b564f62905e4b4374ceda3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:23.802397 9881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b011bbe006e4aeb81b3707296424526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:23.811250 9964 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Bootstrap starting.
I20260925 11:06:23.813167 9965 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Bootstrap starting.
I20260925 11:06:23.815245 9966 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Bootstrap starting.
I20260925 11:06:23.815690 9964 tablet_bootstrap.cc:654] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:23.817008 9964 log.cc:824] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:23.817198 9965 tablet_bootstrap.cc:654] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:23.818427 9965 log.cc:824] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:23.818594 9966 tablet_bootstrap.cc:654] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:23.819738 9966 log.cc:824] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:23.820401 9964 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: No bootstrap required, opened a new log
I20260925 11:06:23.820756 9964 ts_tablet_manager.cc:1397] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 11:06:23.821239 9965 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: No bootstrap required, opened a new log
I20260925 11:06:23.821501 9965 ts_tablet_manager.cc:1397] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 11:06:23.829080 9965 raft_consensus.cc:348] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.829537 9965 raft_consensus.cc:374] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:23.829650 9966 tablet_bootstrap.cc:492] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: No bootstrap required, opened a new log
I20260925 11:06:23.829664 9965 raft_consensus.cc:729] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9624a440cb34d43a8fea57e5acd7274, State: Initialized, Role: FOLLOWER
I20260925 11:06:23.830051 9966 ts_tablet_manager.cc:1397] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260925 11:06:23.830299 9965 consensus_queue.cc:260] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.832648 9965 ts_tablet_manager.cc:1428] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 11:06:23.833488 9964 raft_consensus.cc:348] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.833951 9964 raft_consensus.cc:374] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:23.834052 9964 raft_consensus.cc:729] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c4a77fc0ab4dd19b1b3a60b7100cd8, State: Initialized, Role: FOLLOWER
I20260925 11:06:23.834721 9964 consensus_queue.cc:260] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.836997 9813 heartbeater.cc:503] Master 127.8.181.62:32829 was elected leader, sending a full tablet report...
I20260925 11:06:23.837687 9964 ts_tablet_manager.cc:1428] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260925 11:06:23.840173 9966 raft_consensus.cc:348] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.840546 9966 raft_consensus.cc:374] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:23.840662 9966 raft_consensus.cc:729] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1ae858bca9446a8b96731a88fd504d, State: Initialized, Role: FOLLOWER
I20260925 11:06:23.841355 9966 consensus_queue.cc:260] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.842456 9946 heartbeater.cc:503] Master 127.8.181.62:32829 was elected leader, sending a full tablet report...
I20260925 11:06:23.843127 9966 ts_tablet_manager.cc:1428] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 11:06:23.852288 9970 raft_consensus.cc:482] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:23.852655 9970 raft_consensus.cc:504] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.854604 9970 leader_election.cc:288] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1c4a77fc0ab4dd19b1b3a60b7100cd8 (127.8.181.2:39763), fc1ae858bca9446a8b96731a88fd504d (127.8.181.3:45547)
I20260925 11:06:23.862653 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.854901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:37043 (local address 127.8.181.2:39763)
0925 11:06:23.855129 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.855138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.855706 (+ 568us) server_negotiation.cc:407] Connection header received
0925 11:06:23.855971 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.855980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.856079 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.856172 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:23.856787 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.857775 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.860400 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.861597 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.861747 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.861851 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.862214 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.862370 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:06:23.862723 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.855901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:56405 (local address 127.8.181.3:45547)
0925 11:06:23.856126 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.856133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.856482 (+ 349us) server_negotiation.cc:407] Connection header received
0925 11:06:23.856669 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.856673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.856753 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.856848 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:23.857292 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.858183 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.860550 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.861798 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.861914 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.861984 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.862315 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.862432 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:06:23.862813 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.855769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45547 (local address 127.8.181.1:56405)
0925 11:06:23.856374 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.856397 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.856511 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.856861 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.856864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.856883 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:23.857156 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.857162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.858315 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.858320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.860412 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.860424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.862003 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.862008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.862178 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.862204 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.862276 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.862373 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":4,"thread_start_us":272,"threads_started":1}
I20260925 11:06:23.863242 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.854910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:39763 (local address 127.8.181.1:37043)
0925 11:06:23.855603 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:23.855619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:23.855804 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:23.856224 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:23.856228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:23.856252 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:23.856652 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.856662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.857897 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.857901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.860244 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:23.860259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.861815 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.861819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:23.862033 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.862070 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 11:06:23.862181 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 11:06:23.862332 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":62,"thread_start_us":410,"threads_started":1}
I20260925 11:06:23.863868 9901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b011bbe006e4aeb81b3707296424526" candidate_uuid: "a9624a440cb34d43a8fea57e5acd7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1ae858bca9446a8b96731a88fd504d" is_pre_election: true
I20260925 11:06:23.864117 9768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b011bbe006e4aeb81b3707296424526" candidate_uuid: "a9624a440cb34d43a8fea57e5acd7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" is_pre_election: true
I20260925 11:06:23.864439 9901 raft_consensus.cc:2454] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9624a440cb34d43a8fea57e5acd7274 in term 0.
I20260925 11:06:23.864692 9768 raft_consensus.cc:2454] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9624a440cb34d43a8fea57e5acd7274 in term 0.
I20260925 11:06:23.865167 9568 leader_election.cc:302] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9624a440cb34d43a8fea57e5acd7274, fc1ae858bca9446a8b96731a88fd504d; no voters:
I20260925 11:06:23.865487 9970 raft_consensus.cc:2781] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:23.865687 9970 raft_consensus.cc:482] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:23.865818 9970 raft_consensus.cc:3037] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:23.868033 9970 raft_consensus.cc:504] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.868649 9970 leader_election.cc:288] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [CANDIDATE]: Term 1 election: Requested vote from peers e1c4a77fc0ab4dd19b1b3a60b7100cd8 (127.8.181.2:39763), fc1ae858bca9446a8b96731a88fd504d (127.8.181.3:45547)
I20260925 11:06:23.869151 9768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b011bbe006e4aeb81b3707296424526" candidate_uuid: "a9624a440cb34d43a8fea57e5acd7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8"
I20260925 11:06:23.869203 9901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b011bbe006e4aeb81b3707296424526" candidate_uuid: "a9624a440cb34d43a8fea57e5acd7274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc1ae858bca9446a8b96731a88fd504d"
I20260925 11:06:23.869364 9768 raft_consensus.cc:3037] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:23.869395 9901 raft_consensus.cc:3037] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:23.872021 9768 raft_consensus.cc:2454] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9624a440cb34d43a8fea57e5acd7274 in term 1.
I20260925 11:06:23.872043 9901 raft_consensus.cc:2454] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9624a440cb34d43a8fea57e5acd7274 in term 1.
I20260925 11:06:23.872447 9568 leader_election.cc:302] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9624a440cb34d43a8fea57e5acd7274, fc1ae858bca9446a8b96731a88fd504d; no voters:
I20260925 11:06:23.872748 9970 raft_consensus.cc:2781] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:23.873202 9970 raft_consensus.cc:686] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [term 1 LEADER]: Becoming Leader. State: Replica: a9624a440cb34d43a8fea57e5acd7274, State: Running, Role: LEADER
I20260925 11:06:23.873670 9970 consensus_queue.cc:237] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } }
I20260925 11:06:23.878448 9489 catalog_manager.cc:5495] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9624a440cb34d43a8fea57e5acd7274 (127.8.181.1). New cstate: current_term: 1 leader_uuid: "a9624a440cb34d43a8fea57e5acd7274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9624a440cb34d43a8fea57e5acd7274" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1c4a77fc0ab4dd19b1b3a60b7100cd8" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc1ae858bca9446a8b96731a88fd504d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45547 } health_report { overall_health: UNKNOWN } } }
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:23.908963 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.893253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.8.181.1:33737)
0925 11:06:23.893433 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.893437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.893968 (+ 531us) server_negotiation.cc:407] Connection header received
0925 11:06:23.894070 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.894074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.894145 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.894283 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.895893 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.896716 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.897987 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.898395 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.898562 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.902752 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.902776 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.902804 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.902846 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.906032 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.906850 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.906865 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.906868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.907029 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.907426 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.907430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.907432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.907769 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.908053 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.908372 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.908639 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 11:06:23.925038 9680 tablet.cc:2369] T 2b011bbe006e4aeb81b3707296424526 P a9624a440cb34d43a8fea57e5acd7274: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:23.933604 9814 tablet.cc:2369] T 2b011bbe006e4aeb81b3707296424526 P e1c4a77fc0ab4dd19b1b3a60b7100cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260925 11:06:23.947463 9947 tablet.cc:2369] T 2b011bbe006e4aeb81b3707296424526 P fc1ae858bca9446a8b96731a88fd504d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:23.955420 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.941358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.8.181.1:33737)
0925 11:06:23.941589 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.941595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.941831 (+ 236us) server_negotiation.cc:407] Connection header received
0925 11:06:23.942024 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.942028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.942100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.942194 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.943831 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.944595 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.945927 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.946337 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.946432 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.950431 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.950446 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.950449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.950534 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.952967 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.953559 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.953563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.953565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.953678 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.954026 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.954029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.954031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.954294 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.954569 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.954858 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.955024 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 11:06:23 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:24.000100 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:23.986063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.8.181.1:33737)
0925 11:06:23.986325 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:23.986334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:23.986455 (+ 121us) server_negotiation.cc:407] Connection header received
0925 11:06:23.986748 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:23.986752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:23.986829 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:23.986915 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:23.988294 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.989032 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.990328 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:23.990788 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:23.990881 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:23.994834 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:23.994839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:23.994842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:23.994877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:23.997487 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.998156 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.998160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.998162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.998278 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:23.998689 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:23.998692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:23.998694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:23.998949 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:23.999158 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:23.999512 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 11:06:23.999759 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260925 11:06:24.001878 9594 tablet_service.cc:2908] Rejecting scan request for tablet 2b011bbe006e4aeb81b3707296424526: Uninitialized: clean time has not yet been initialized
I20260925 11:06:24.005417 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9551
I20260925 11:06:24.019124 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:24.006401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.8.181.1:33737)
0925 11:06:24.006654 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:24.006664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:24.007044 (+ 380us) server_negotiation.cc:407] Connection header received
0925 11:06:24.007196 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:24.007200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:24.007274 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:24.007359 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:24.008640 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.009330 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.010711 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.011119 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.011214 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:24.013664 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:24.013668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:24.013671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:24.013703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:24.016317 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.017091 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.017100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.017103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.017244 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.017710 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.017713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.017715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.017967 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:24.018149 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:24.018450 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 11:06:24.018753 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 11:06:24.019852 9674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:24.112486 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9551
I20260925 11:06:24.131759 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9685
Sep 25 11:06:24 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:24.146708 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:24.132825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.8.181.2:39763)
0925 11:06:24.133007 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:24.133012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:24.133028 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:06:24.133077 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:24.133081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:24.133138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:24.133258 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:24.134557 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.135423 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.136768 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.137144 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.137270 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:24.141221 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:24.141233 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:24.141249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:24.141282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:24.143952 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.144627 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.144634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.144637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.144766 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.145170 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.145177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.145182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.145545 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:24.145873 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:24.146135 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 11:06:24.146345 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:06:24.147418 9808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:24.235865 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9685
I20260925 11:06:24.252593 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9818
Sep 25 11:06:24 dist-test-slave-6042 krb5kdc[9445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:24.268405 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:24.253475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.8.181.3:45547)
0925 11:06:24.253646 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:24.253650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:24.253668 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:06:24.253829 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:24.253834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:24.253897 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:24.253981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:24.255289 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.256000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.257438 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:24.257837 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:24.257928 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:24.262143 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:24.262159 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:24.262177 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:24.262212 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:24.265481 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.266161 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.266168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.266171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.266345 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:24.266917 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:24.266921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:24.266923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:24.267286 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:24.267531 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:24.267850 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 11:06:24.268030 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:06:24.269090 9941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:24.360360 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9818
I20260925 11:06:24.378269 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 9458
I20260925 11:06:24.379487 9520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:24.489590 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 9458
2026-09-25T11:06:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4159 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:24 dist-test-slave-6042 krb5kdc[9994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:24 dist-test-slave-6042 krb5kdc[9994](info): set up 2 sockets
Sep 25 11:06:24 dist-test-slave-6042 krb5kdc[9994](info): commencing operation
krb5kdc: starting...
W20260925 11:06:25.583109 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:25 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:25.783128 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35787
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:26.111979 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:26.112375 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:26.112556 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:26.119910 10007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:26.120086 10007 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 11:06:26.120167 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:26.120261 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:26.120360 10007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:26.120451 10007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:26.130453 10007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:35787
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:26.131850 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:26.133224 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:26.145540 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:26.145493 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:26.145565 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:26.146193 10007 server_base.cc:1034] running on GCE node
I20260925 11:06:26.147054 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:26.148515 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:26.149739 10007 hybrid_clock.cc:648] HybridClock initialized: now 1790334386149680 us; error 33 us; skew 500 ppm
I20260925 11:06:26.150084 10007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:26 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334386, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:26.153421 10007 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:26.155089 10007 webserver.cc:460] Webserver started at http://127.8.181.62:41631/ using document root <none> and password file <none>
I20260925 11:06:26.155831 10007 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:26.155947 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:26.156306 10007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:26.159538 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "22a82e879e33494fb100c8df3f20d0dc"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "94359c111810e8b1cc1af567ece35c02"
server_key_iv: "c3fa8b0b865d3358dd7cd99c6a5629a2"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.160401 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "22a82e879e33494fb100c8df3f20d0dc"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "94359c111810e8b1cc1af567ece35c02"
server_key_iv: "c3fa8b0b865d3358dd7cd99c6a5629a2"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.165887 10007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:26.169776 10022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:26.171368 10007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:26.171582 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "22a82e879e33494fb100c8df3f20d0dc"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "94359c111810e8b1cc1af567ece35c02"
server_key_iv: "c3fa8b0b865d3358dd7cd99c6a5629a2"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.171829 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:26.190429 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:26.195029 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:26.195341 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:26.218017 10007 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:35787
I20260925 11:06:26.218025 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:35787 every 8 connection(s)
I20260925 11:06:26.219527 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:26.222668 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10007
I20260925 11:06:26.222899 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:26.223724 8916 external_mini_cluster.cc:1428] Setting key be1fb63b323ac29be630df4dc6c97628
I20260925 11:06:26.223798 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:26.239091 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Bootstrap starting.
Sep 25 11:06:26 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:26.242964 10084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:26.244451 10084 log.cc:824] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:26.247637 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: No bootstrap required, opened a new log
I20260925 11:06:26.250059 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:26.225803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.8.181.62:35787)
0925 11:06:26.226696 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:26.226707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:26.226751 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:06:26.228627 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:26.228650 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:26.229052 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:26.229419 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:26.231333 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.232903 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.234733 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.235599 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.235849 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:26.241183 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:26.241195 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:26.241214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:26.241270 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:26.245444 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.246217 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.246224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.246231 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.246414 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.246850 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.246854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.246856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.247363 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:26.247702 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:26.248150 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 11:06:26.248647 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":402,"threads_started":1}
I20260925 11:06:26.256793 10084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } }
I20260925 11:06:26.257200 10084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:26.257268 10084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a82e879e33494fb100c8df3f20d0dc, State: Initialized, Role: FOLLOWER
I20260925 11:06:26.257865 10084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } }
I20260925 11:06:26.258111 10084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:26.258250 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:26.258498 10084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:26.260870 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } }
I20260925 11:06:26.261421 10084 leader_election.cc:302] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22a82e879e33494fb100c8df3f20d0dc; no voters:
I20260925 11:06:26.262027 10084 leader_election.cc:288] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:26.262172 10089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:26.263667 10089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [term 1 LEADER]: Becoming Leader. State: Replica: 22a82e879e33494fb100c8df3f20d0dc, State: Running, Role: LEADER
I20260925 11:06:26.264379 10084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:26.264204 10089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } }
I20260925 11:06:26.267203 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22a82e879e33494fb100c8df3f20d0dc. Latest consensus state: current_term: 1 leader_uuid: "22a82e879e33494fb100c8df3f20d0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } } }
I20260925 11:06:26.267575 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:26.268033 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22a82e879e33494fb100c8df3f20d0dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a82e879e33494fb100c8df3f20d0dc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35787 } } }
I20260925 11:06:26.268296 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:26.270068 10098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:26.275535 10098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:26.285337 10098 catalog_manager.cc:1347] Generated new cluster ID: 228f33303cf44dcb8c240eb25df098d6
I20260925 11:06:26.285473 10098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:26.323022 10098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:26.324422 10098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:26.343377 10098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Generated new TSK 0
I20260925 11:06:26.344340 10098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:26.403538 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35787
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:26.742424 10110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:26.742781 10110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:26.742954 10110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:26.750619 10110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:26.750769 10110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:26.750921 10110 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:26.751066 10110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:26.761106 10110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:26.762672 10110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:26.764014 10110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:26.782889 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:26.782958 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:26.782912 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:26.783977 10110 server_base.cc:1034] running on GCE node
I20260925 11:06:26.784680 10110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:26.785756 10110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:26.787061 10110 hybrid_clock.cc:648] HybridClock initialized: now 1790334386786968 us; error 70 us; skew 500 ppm
I20260925 11:06:26.787395 10110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:26 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334386, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:26.790879 10110 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:26.792439 10110 webserver.cc:460] Webserver started at http://127.8.181.1:37587/ using document root <none> and password file <none>
I20260925 11:06:26.793160 10110 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:26.793295 10110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:26.793627 10110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:26.796608 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "7c49f21b91b6752e70a1ab597ec0a791"
server_key_iv: "15e6fcf868278dedad47b7cb863b317e"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.797406 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "7c49f21b91b6752e70a1ab597ec0a791"
server_key_iv: "15e6fcf868278dedad47b7cb863b317e"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.802713 10110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 11:06:26.806123 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:26.807578 10110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:26.807773 10110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
format_stamp: "Formatted at 2026-09-25 11:06:26 on dist-test-slave-6042"
server_key: "7c49f21b91b6752e70a1ab597ec0a791"
server_key_iv: "15e6fcf868278dedad47b7cb863b317e"
server_key_version: "encryptionkey@0"
I20260925 11:06:26.808009 10110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:26.827654 10110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:26.831967 10110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:26.832281 10110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:26.835179 10110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:26.835307 10110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:26.835482 10110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:26.835551 10110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:26 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334386, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:26.859244 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:26.835661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:53931 (local address 127.8.181.62:35787)
0925 11:06:26.836494 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:26.836501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:26.837490 (+ 989us) server_negotiation.cc:407] Connection header received
0925 11:06:26.839711 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:26.839719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:26.839823 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:26.839943 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:26.842271 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.843372 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.846297 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.846825 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.846965 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:26.851718 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:26.851725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:26.851736 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:26.851799 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:26.855246 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.856440 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.856444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.856446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.856563 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.857152 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.857156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.857158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.857409 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:26.857510 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:26.858733 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0925 11:06:26.858988 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":602,"threads_started":1}
I20260925 11:06:26.861305 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:26.836380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.1:53931)
0925 11:06:26.837293 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:26.837342 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:26.839402 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:26.840226 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:26.840241 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:26.841150 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:26.842018 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:26.842047 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.843693 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.843705 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:26.846137 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:26.846148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.847070 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.847077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:26.847308 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:26.848056 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:26.848097 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:26.851441 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:26.855458 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:26.855472 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:26.855497 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:26.856298 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:26.856686 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:26.856693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:26.856699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:26.857028 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:26.857550 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:26.857563 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:26.858198 (+ 635us) client_negotiation.cc:769] Sending connection context
0925 11:06:26.858636 (+ 438us) client_negotiation.cc:240] Negotiation successful
0925 11:06:26.859374 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":407,"threads_started":1}
I20260925 11:06:26.887509 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:26.868861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.1:51089)
0925 11:06:26.869590 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:26.869617 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:26.869809 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:26.870256 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:26.870261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:26.871029 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:26.871398 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:26.871411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.873169 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.873178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:26.875642 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:26.875680 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.877217 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.877224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:26.877467 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:26.878438 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:26.878509 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:26.879764 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:26.884237 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:26.884243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:26.884246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:26.884842 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:26.885391 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:26.885397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:26.885401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:26.885555 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:26.886235 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:26.886239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:26.886391 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 11:06:26.886552 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 11:06:26.886939 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":389,"threads_started":1}
I20260925 11:06:26.888208 10110 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34363
I20260925 11:06:26.888289 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34363 every 8 connection(s)
I20260925 11:06:26.888665 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:26.868941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:51089 (local address 127.8.181.62:35787)
0925 11:06:26.869177 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:26.869186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:26.869716 (+ 530us) server_negotiation.cc:407] Connection header received
0925 11:06:26.869966 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:26.869975 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:26.870083 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:26.870204 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:26.871557 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.872959 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.875884 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:26.876697 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:26.876848 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:26.880066 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:26.880076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:26.880082 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:26.880126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:26.884077 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.884996 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.885004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.885011 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.885187 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:26.885691 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:26.885699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:26.885705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:26.886117 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:26.886267 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:26.888105 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0925 11:06:26.888378 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:06:26.889900 10110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:26.892315 10246 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35787
I20260925 11:06:26.892554 10246 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:26.893216 10246 heartbeater.cc:511] Master 127.8.181.62:35787 requested a full tablet report, sending...
I20260925 11:06:26.894879 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10110
I20260925 11:06:26.895100 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:26.895134 10039 ts_manager.cc:194] Registered new tserver with Master: ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363)
I20260925 11:06:26.895769 8916 external_mini_cluster.cc:1428] Setting key 5663d831bb9c5f045a8b817354ea8dbb
I20260925 11:06:26.896759 10039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:53931
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:26.939571 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35787
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:27.267616 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:27.267957 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:27.268113 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:27.276155 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:27.276317 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:27.276494 10250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:27.276654 10250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:27.286799 10250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:27.288151 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:27.289477 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:27.303614 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:27.303725 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:27.303607 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:27.304499 10250 server_base.cc:1034] running on GCE node
I20260925 11:06:27.305087 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:27.306061 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:27.307317 10250 hybrid_clock.cc:648] HybridClock initialized: now 1790334387307238 us; error 62 us; skew 500 ppm
I20260925 11:06:27.307668 10250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:27 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:27.311084 10250 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:27.312700 10250 webserver.cc:460] Webserver started at http://127.8.181.2:43789/ using document root <none> and password file <none>
I20260925 11:06:27.313374 10250 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:27.313459 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:27.313732 10250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:27.316666 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "2007ffda629b453f98850afcc1d1f081"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "76bfa80fe8bca4fd375f8c170ef7eda8"
server_key_iv: "7b87f9ffe705a37f6d2011f286c73e14"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.317479 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "2007ffda629b453f98850afcc1d1f081"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "76bfa80fe8bca4fd375f8c170ef7eda8"
server_key_iv: "7b87f9ffe705a37f6d2011f286c73e14"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.322573 10250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 11:06:27.326015 10265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:27.327571 10250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:27.327725 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "2007ffda629b453f98850afcc1d1f081"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "76bfa80fe8bca4fd375f8c170ef7eda8"
server_key_iv: "7b87f9ffe705a37f6d2011f286c73e14"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.327914 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:27.356885 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:27.361901 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:27.362250 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:27.366317 10250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:27.366437 10250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:27.366631 10250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:27.366720 10250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:27 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:27.393929 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.366890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:59423 (local address 127.8.181.62:35787)
0925 11:06:27.367161 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:27.367169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:27.369066 (+ 1897us) server_negotiation.cc:407] Connection header received
0925 11:06:27.372178 (+ 3112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:27.372186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:27.372328 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:27.372584 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:27.374762 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.375867 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.378247 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.378899 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.379083 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.385493 (+ 6410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:27.385503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:27.385510 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:27.385569 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:27.389909 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.390877 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.390883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.390888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.391056 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.391720 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.391728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.391734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.392215 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:27.392389 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:27.393176 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 11:06:27.393507 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 11:06:27.395323 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.367690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.2:59423)
0925 11:06:27.368814 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:27.368864 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:27.371770 (+ 2906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:27.373055 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:27.373066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:27.373710 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:27.374433 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.374455 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.376090 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.376099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.378006 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.378046 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.379291 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.379304 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.379787 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.380971 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:27.381020 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:27.385178 (+ 4158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:27.390124 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.390132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.390147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.390729 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.391196 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.391208 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.391213 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.391539 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.392383 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:27.392393 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:27.392775 (+ 382us) client_negotiation.cc:769] Sending connection context
0925 11:06:27.393005 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 11:06:27.393541 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":456,"threads_started":1}
I20260925 11:06:27.426954 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.406672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:49761 (local address 127.8.181.62:35787)
0925 11:06:27.406900 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:27.406909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:27.411367 (+ 4458us) server_negotiation.cc:407] Connection header received
0925 11:06:27.411653 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:27.411660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:27.411762 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:27.411905 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:27.413525 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.414878 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.417082 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.417678 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.417816 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.420282 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:27.420288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:27.420291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:27.420323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:27.423466 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.424594 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.424598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.424600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.424786 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.425232 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.425235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.425237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.425702 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:27.425863 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:27.426302 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 11:06:27.426577 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:06:27.427105 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.406227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.2:49761)
0925 11:06:27.411210 (+ 4983us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:27.411235 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:27.411468 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:27.411946 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:27.411951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:27.412919 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:27.413371 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.413382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.415072 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.415079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.416853 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.416874 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.417898 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.417904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.418066 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.418854 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:27.418872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:27.420037 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:27.423704 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.423711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.423716 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.424374 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.424946 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.424955 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.424959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.425104 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.425825 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:27.425830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:27.426021 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 11:06:27.426198 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 11:06:27.426582 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4791,"thread_start_us":4610,"threads_started":1}
I20260925 11:06:27.434448 10250 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:46593
I20260925 11:06:27.434580 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:46593 every 8 connection(s)
I20260925 11:06:27.435837 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:27.437970 10385 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35787
I20260925 11:06:27.438198 10385 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:27.438938 10385 heartbeater.cc:511] Master 127.8.181.62:35787 requested a full tablet report, sending...
I20260925 11:06:27.440176 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10250
I20260925 11:06:27.440400 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:27.440397 10038 ts_manager.cc:194] Registered new tserver with Master: 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593)
I20260925 11:06:27.440913 8916 external_mini_cluster.cc:1428] Setting key 5c958225c2968ed71d75a63d24ddc782
I20260925 11:06:27.441331 10038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:59423
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:27.486116 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35787
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:27.819958 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:27.820286 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:27.820470 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:27.828229 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:27.828361 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:27.828537 10389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:27.828691 10389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:27.839613 10389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:35787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:27.840939 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:27.842218 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:27.856207 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:27.856271 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:27.856397 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:27.857255 10389 server_base.cc:1034] running on GCE node
I20260925 11:06:27.857949 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:27.858983 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:27.860231 10389 hybrid_clock.cc:648] HybridClock initialized: now 1790334387860173 us; error 40 us; skew 500 ppm
I20260925 11:06:27.860539 10389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:27 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:27.864010 10389 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:27.865679 10389 webserver.cc:460] Webserver started at http://127.8.181.3:33363/ using document root <none> and password file <none>
I20260925 11:06:27.866381 10389 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:27.866516 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:27.866854 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:27.870270 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "06eb478d88ba49e0a14fb307607eae4f"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "408708f175e176edcd107f09575a9a84"
server_key_iv: "36998be862f68b554b2a1eaf47be4a09"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.871229 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "06eb478d88ba49e0a14fb307607eae4f"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "408708f175e176edcd107f09575a9a84"
server_key_iv: "36998be862f68b554b2a1eaf47be4a09"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.876814 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:27.880434 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:27.881886 10389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:27.882063 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "06eb478d88ba49e0a14fb307607eae4f"
format_stamp: "Formatted at 2026-09-25 11:06:27 on dist-test-slave-6042"
server_key: "408708f175e176edcd107f09575a9a84"
server_key_iv: "36998be862f68b554b2a1eaf47be4a09"
server_key_version: "encryptionkey@0"
I20260925 11:06:27.882277 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:27.895807 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:27.899541 10246 heartbeater.cc:503] Master 127.8.181.62:35787 was elected leader, sending a full tablet report...
I20260925 11:06:27.900363 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:27.900637 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:27.903411 10389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:27.903523 10389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:27.903759 10389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:27.903844 10389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:27 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:27.927817 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.903817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:34137 (local address 127.8.181.62:35787)
0925 11:06:27.904003 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:27.904009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:27.905489 (+ 1480us) server_negotiation.cc:407] Connection header received
0925 11:06:27.907622 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:27.907629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:27.907733 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:27.907868 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:27.910396 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.911579 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.914677 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.915310 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.915479 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.920720 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:27.920729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:27.920739 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:27.920774 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:27.923874 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.924952 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.924962 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.924968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.925172 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.925847 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.925855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.925858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.926255 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:27.926429 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:27.927202 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 11:06:27.927434 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:06:27.928855 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.904444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.3:34137)
0925 11:06:27.905294 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:27.905346 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:27.907315 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:27.908165 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:27.908179 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:27.909156 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:27.910171 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.910210 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.911776 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.911782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.914390 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.914404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.915825 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.915830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.916145 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.917045 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:27.917088 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:27.920443 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:27.924033 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.924049 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.924074 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.924765 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.925331 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.925339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.925345 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.925684 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.926391 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:27.926400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:27.926820 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 11:06:27.927083 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 11:06:27.927666 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":366,"threads_started":1}
I20260925 11:06:27.955338 10389 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:46333
I20260925 11:06:27.955482 10523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:46333 every 8 connection(s)
I20260925 11:06:27.956898 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:27.959374 10524 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35787
I20260925 11:06:27.959645 10524 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:27.960496 10524 heartbeater.cc:511] Master 127.8.181.62:35787 requested a full tablet report, sending...
I20260925 11:06:27.961740 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.948237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:36643 (local address 127.8.181.62:35787)
0925 11:06:27.948479 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:27.948488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:27.949290 (+ 802us) server_negotiation.cc:407] Connection header received
0925 11:06:27.949504 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:27.949509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:27.949592 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:27.949712 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:27.950898 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.951624 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.952939 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.953351 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.953495 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.955648 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:27.955652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:27.955656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:27.955687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:27.958855 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.959635 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.959641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.959645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.959845 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:27.960245 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:27.960249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:27.960251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:27.960536 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:27.960750 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:27.961159 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 11:06:27.961379 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:06:27.961856 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:27.948226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.8.181.3:36643)
0925 11:06:27.949140 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:27.949168 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:27.949358 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:27.949738 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:27.949741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:27.950218 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:27.950667 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.950684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.951768 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.951774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.952792 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:27.952803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:27.953537 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:27.953544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:27.953720 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:27.954366 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:27.954382 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:27.955399 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:27.959000 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.959006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.959010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.959512 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.959960 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:27.959965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:27.959968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:27.960137 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:27.960714 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:27.960721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:27.960900 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 11:06:27.961073 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 11:06:27.961439 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":384,"threads_started":1}
I20260925 11:06:27.962241 10038 ts_manager.cc:194] Registered new tserver with Master: 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333)
I20260925 11:06:27.963483 10038 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:34137
I20260925 11:06:27.965803 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10389
I20260925 11:06:27.965992 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:27.966411 8916 external_mini_cluster.cc:1428] Setting key 6aad22db5fcb5cc7e73a55237d70b0ae
I20260925 11:06:27.969347 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 11:06:27 dist-test-slave-6042 krb5kdc[9994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 11:06:28 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:28.017611 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.002285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.8.181.62:35787)
0925 11:06:28.002562 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.002570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.002803 (+ 233us) server_negotiation.cc:407] Connection header received
0925 11:06:28.003037 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.003041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.003122 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.003254 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:28.004769 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.005549 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.006955 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.007393 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.007546 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.011882 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:28.011888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:28.011892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:28.011934 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:28.014788 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:28.015544 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:28.015548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:28.015551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:28.015684 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:28.016106 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:28.016109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:28.016111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:28.016394 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:28.016673 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.016978 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.017242 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 11:06:28.022559 10038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:28.029613 10038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:28.066776 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.043460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.8.181.1:34363)
0925 11:06:28.056636 (+ 13176us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.056651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.057506 (+ 855us) server_negotiation.cc:407] Connection header received
0925 11:06:28.058792 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.058801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.059041 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.059221 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.060137 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.061569 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.064187 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.065646 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.065782 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.065973 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.066265 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.066427 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12942,"thread_start_us":13194,"threads_started":1}
I20260925 11:06:28.066931 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.042960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34363 (local address 127.0.0.1:42238)
0925 11:06:28.057353 (+ 14393us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.057375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.058583 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.059448 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.059452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.059485 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.059929 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.059942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.061757 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.061762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.063999 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.064010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.065807 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.065813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.066015 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.066077 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.066224 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.066423 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14238,"mutex_wait_us":46,"thread_start_us":10830,"threads_started":1}
I20260925 11:06:28.067036 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.042901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46593 (local address 127.0.0.1:37424)
0925 11:06:28.053179 (+ 10278us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.053250 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.053530 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.058570 (+ 5040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.058591 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.058632 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.059005 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.059014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.060770 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.060776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.063355 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.063370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.066058 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.066065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.066276 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.066306 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.066380 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.066551 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9854,"mutex_wait_us":3,"thread_start_us":14712,"threads_started":1}
I20260925 11:06:28.067727 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.042926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.8.181.2:46593)
0925 11:06:28.057606 (+ 14680us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.057614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.057638 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:06:28.057745 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.057755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.058069 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.058292 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.059180 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.060604 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.063535 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.065518 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.065643 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.065898 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.067262 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.067433 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14509,"thread_start_us":14407,"threads_started":1}
I20260925 11:06:28.067533 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.055184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46333 (local address 127.0.0.1:33490)
0925 11:06:28.058522 (+ 3338us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.058539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.058663 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.059217 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.059221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.059251 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.059685 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.059698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.061815 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.061822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.064096 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.064111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.066647 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.066653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.066853 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.066895 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.066988 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.067157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3174,"mutex_wait_us":134,"spinlock_wait_cycles":4480,"thread_start_us":2329,"threads_started":1}
I20260925 11:06:28.068182 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.055386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.8.181.3:46333)
0925 11:06:28.058394 (+ 3008us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.058402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.058776 (+ 374us) server_negotiation.cc:407] Connection header received
0925 11:06:28.058863 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.058869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.059041 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.059221 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.059909 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.061646 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.064299 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.066121 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.066241 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.067580 (+ 1339us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.067658 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.067840 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2844,"thread_start_us":2666,"threads_started":1}
I20260925 11:06:28.071238 10315 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56709f328848d5b405d02233d6db06 (DEFAULT_TABLE table=test-table [id=46474e795df44cdd9b6ceeeabce403f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.071270 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56709f328848d5b405d02233d6db06 (DEFAULT_TABLE table=test-table [id=46474e795df44cdd9b6ceeeabce403f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.071616 10454 tablet_service.cc:1467] Processing CreateTablet for tablet 9b56709f328848d5b405d02233d6db06 (DEFAULT_TABLE table=test-table [id=46474e795df44cdd9b6ceeeabce403f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.072553 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56709f328848d5b405d02233d6db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.072664 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56709f328848d5b405d02233d6db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.072889 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b56709f328848d5b405d02233d6db06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.084721 10543 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Bootstrap starting.
I20260925 11:06:28.087307 10544 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Bootstrap starting.
I20260925 11:06:28.090021 10543 tablet_bootstrap.cc:654] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.090533 10545 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Bootstrap starting.
I20260925 11:06:28.091156 10544 tablet_bootstrap.cc:654] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.091514 10543 log.cc:824] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:28.092381 10544 log.cc:824] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:28.093922 10545 tablet_bootstrap.cc:654] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.095095 10545 log.cc:824] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:28.095160 10543 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: No bootstrap required, opened a new log
I20260925 11:06:28.095461 10543 ts_tablet_manager.cc:1397] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260925 11:06:28.097272 10544 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: No bootstrap required, opened a new log
I20260925 11:06:28.097574 10544 ts_tablet_manager.cc:1397] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 11:06:28.098129 10545 tablet_bootstrap.cc:492] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: No bootstrap required, opened a new log
I20260925 11:06:28.098405 10545 ts_tablet_manager.cc:1397] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 11:06:28.106841 10545 raft_consensus.cc:348] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.107277 10545 raft_consensus.cc:374] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.107393 10545 raft_consensus.cc:729] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06eb478d88ba49e0a14fb307607eae4f, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.107517 10543 raft_consensus.cc:348] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.108037 10543 raft_consensus.cc:374] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.108222 10545 consensus_queue.cc:260] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.108466 10544 raft_consensus.cc:348] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.108945 10544 raft_consensus.cc:374] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.109129 10544 raft_consensus.cc:729] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8d8154eb2c43b3869f1a66cb1c6945, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.109407 10524 heartbeater.cc:503] Master 127.8.181.62:35787 was elected leader, sending a full tablet report...
I20260925 11:06:28.108198 10543 raft_consensus.cc:729] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2007ffda629b453f98850afcc1d1f081, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.109838 10544 consensus_queue.cc:260] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.110217 10545 ts_tablet_manager.cc:1428] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20260925 11:06:28.110411 10543 consensus_queue.cc:260] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.112020 10544 ts_tablet_manager.cc:1428] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260925 11:06:28.113116 10385 heartbeater.cc:503] Master 127.8.181.62:35787 was elected leader, sending a full tablet report...
I20260925 11:06:28.113905 10543 ts_tablet_manager.cc:1428] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
W20260925 11:06:28.142928 10247 tablet.cc:2369] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:28.187963 10386 tablet.cc:2369] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:28.209223 10525 tablet.cc:2369] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:28.319216 10551 raft_consensus.cc:482] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:28.319613 10551 raft_consensus.cc:504] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.321367 10551 leader_election.cc:288] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333)
I20260925 11:06:28.329113 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.321876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:60153 (local address 127.8.181.1:34363)
0925 11:06:28.322071 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.322076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.322733 (+ 657us) server_negotiation.cc:407] Connection header received
0925 11:06:28.322942 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.322948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.323035 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.323143 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.323794 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.324794 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.327031 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.328207 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.328309 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.328386 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.328778 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.328875 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 11:06:28.329280 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.321884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34363 (local address 127.8.181.2:60153)
0925 11:06:28.322618 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.322640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.322783 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.323244 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.323250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.323284 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.323621 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.323634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.324939 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.324944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.326866 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.326885 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.328418 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.328424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.328602 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.328635 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.328734 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.328875 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":384,"threads_started":1}
I20260925 11:06:28.330216 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.322756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46333 (local address 127.8.181.2:41679)
0925 11:06:28.323417 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.323451 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.323573 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.323909 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.323914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.323970 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.324351 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.324363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.325785 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.325789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.327862 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.327876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.329530 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.329538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.329726 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.329765 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.329864 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.330002 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":352,"threads_started":1}
I20260925 11:06:28.330348 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.322921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:41679 (local address 127.8.181.3:46333)
0925 11:06:28.323136 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.323141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.323559 (+ 418us) server_negotiation.cc:407] Connection header received
0925 11:06:28.323698 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.323706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.323792 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.323859 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.324552 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.325619 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.328023 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.329121 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.329243 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.329365 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.329925 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.330115 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:06:28.330610 10201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56709f328848d5b405d02233d6db06" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" is_pre_election: true
I20260925 11:06:28.331218 10201 raft_consensus.cc:2454] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 0.
I20260925 11:06:28.331444 10475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56709f328848d5b405d02233d6db06" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06eb478d88ba49e0a14fb307607eae4f" is_pre_election: true
I20260925 11:06:28.331905 10269 leader_election.cc:302] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2007ffda629b453f98850afcc1d1f081, ff8d8154eb2c43b3869f1a66cb1c6945; no voters:
I20260925 11:06:28.332060 10475 raft_consensus.cc:2454] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 0.
I20260925 11:06:28.332372 10551 raft_consensus.cc:2781] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:28.332522 10551 raft_consensus.cc:482] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:28.332656 10551 raft_consensus.cc:3037] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.334928 10551 raft_consensus.cc:504] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.335500 10551 leader_election.cc:288] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 election: Requested vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333)
I20260925 11:06:28.336030 10201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56709f328848d5b405d02233d6db06" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
I20260925 11:06:28.336103 10475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b56709f328848d5b405d02233d6db06" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06eb478d88ba49e0a14fb307607eae4f"
I20260925 11:06:28.336241 10201 raft_consensus.cc:3037] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.336334 10475 raft_consensus.cc:3037] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.338539 10201 raft_consensus.cc:2454] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 1.
I20260925 11:06:28.338634 10475 raft_consensus.cc:2454] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 1.
I20260925 11:06:28.338995 10269 leader_election.cc:302] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2007ffda629b453f98850afcc1d1f081, ff8d8154eb2c43b3869f1a66cb1c6945; no voters:
I20260925 11:06:28.339303 10551 raft_consensus.cc:2781] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:28.339748 10551 raft_consensus.cc:686] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 1 LEADER]: Becoming Leader. State: Replica: 2007ffda629b453f98850afcc1d1f081, State: Running, Role: LEADER
I20260925 11:06:28.340242 10551 consensus_queue.cc:237] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.345115 10037 catalog_manager.cc:5495] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2007ffda629b453f98850afcc1d1f081 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "2007ffda629b453f98850afcc1d1f081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } health_report { overall_health: UNKNOWN } } }
I20260925 11:06:28.373389 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.367248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.8.181.2:46593)
0925 11:06:28.367480 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.367487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.367668 (+ 181us) server_negotiation.cc:407] Connection header received
0925 11:06:28.368072 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.368076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.368171 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.368322 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:28.368697 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.369452 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.371114 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.371520 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.371656 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.372176 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:28.372718 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:28.372877 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.373023 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.373114 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:06:28.382285 10201 raft_consensus.cc:1261] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Refusing update from remote peer 2007ffda629b453f98850afcc1d1f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.382319 10475 raft_consensus.cc:1261] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Refusing update from remote peer 2007ffda629b453f98850afcc1d1f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.383311 10554 consensus_queue.cc:1035] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:28.383725 10551 consensus_queue.cc:1035] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:28.393690 10557 mvcc.cc:205] Tried to move back new op lower bound from 7333209654803718144 to 7333209654645444608. Current Snapshot: MvccSnapshot[applied={T|T < 7333209654803718144}]
I20260925 11:06:28.394336 10558 mvcc.cc:205] Tried to move back new op lower bound from 7333209654803718144 to 7333209654645444608. Current Snapshot: MvccSnapshot[applied={T|T < 7333209654803718144}]
I20260925 11:06:28.421553 10037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:28.421860 10037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:28.425746 10037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Sending DeleteTablet for 3 replicas of tablet 9b56709f328848d5b405d02233d6db06
I20260925 11:06:28.427045 10315 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b56709f328848d5b405d02233d6db06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:28 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:37424
I20260925 11:06:28.427119 10454 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b56709f328848d5b405d02233d6db06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:28 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260925 11:06:28.427847 10176 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b56709f328848d5b405d02233d6db06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:28 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:42238
I20260925 11:06:28.431195 10569 tablet_replica.cc:330] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: stopping tablet replica
I20260925 11:06:28.431851 10569 raft_consensus.cc:2229] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:28.432546 10569 raft_consensus.cc:2258] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:28.431862 10037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 11:06:28.433404 10568 tablet_replica.cc:330] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: stopping tablet replica
I20260925 11:06:28.433929 10568 raft_consensus.cc:2229] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 11:06:28.434113 10037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:28.434530 10568 raft_consensus.cc:2258] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:28.436673 10568 ts_tablet_manager.cc:1905] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:28.440062 10567 tablet_replica.cc:330] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: stopping tablet replica
I20260925 11:06:28.440577 10567 raft_consensus.cc:2229] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:28.441210 10567 raft_consensus.cc:2258] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:28.444375 10568 ts_tablet_manager.cc:1918] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:28.444520 10569 ts_tablet_manager.cc:1905] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:28.444624 10568 log.cc:1196] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/9b56709f328848d5b405d02233d6db06
I20260925 11:06:28.445266 10568 ts_tablet_manager.cc:1939] T 9b56709f328848d5b405d02233d6db06 P 06eb478d88ba49e0a14fb307607eae4f: Deleting consensus metadata
I20260925 11:06:28.446521 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 (DEFAULT_TABLE table=test-table [id=6527f1c8b5a347a8863bd4ec0118a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.447067 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442c0521f7446f8900acddf7e0b4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.447396 10026 catalog_manager.cc:4841] TS 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333): tablet 9b56709f328848d5b405d02233d6db06 (table test-table [id=46474e795df44cdd9b6ceeeabce403f5]) successfully deleted
I20260925 11:06:28.451109 10315 tablet_service.cc:1467] Processing CreateTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 (DEFAULT_TABLE table=test-table [id=6527f1c8b5a347a8863bd4ec0118a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.451679 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442c0521f7446f8900acddf7e0b4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.452126 10569 ts_tablet_manager.cc:1918] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:28.452301 10569 log.cc:1196] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/9b56709f328848d5b405d02233d6db06
I20260925 11:06:28.452853 10569 ts_tablet_manager.cc:1939] T 9b56709f328848d5b405d02233d6db06 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting consensus metadata
I20260925 11:06:28.453802 10544 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Bootstrap starting.
I20260925 11:06:28.455395 10454 tablet_service.cc:1467] Processing CreateTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 (DEFAULT_TABLE table=test-table [id=6527f1c8b5a347a8863bd4ec0118a617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:28.455686 10544 tablet_bootstrap.cc:654] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.455938 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7442c0521f7446f8900acddf7e0b4d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.457013 10026 catalog_manager.cc:4841] TS ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363): tablet 9b56709f328848d5b405d02233d6db06 (table test-table [id=46474e795df44cdd9b6ceeeabce403f5]) successfully deleted
I20260925 11:06:28.458561 10544 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: No bootstrap required, opened a new log
I20260925 11:06:28.458781 10544 ts_tablet_manager.cc:1397] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260925 11:06:28.459745 10544 raft_consensus.cc:348] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.460160 10544 raft_consensus.cc:374] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.460321 10544 raft_consensus.cc:729] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8d8154eb2c43b3869f1a66cb1c6945, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.460680 10544 consensus_queue.cc:260] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.461555 10544 ts_tablet_manager.cc:1428] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:28.462934 10543 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Bootstrap starting.
I20260925 11:06:28.463032 10545 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Bootstrap starting.
I20260925 11:06:28.464838 10545 tablet_bootstrap.cc:654] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.464949 10543 tablet_bootstrap.cc:654] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.465939 10567 ts_tablet_manager.cc:1905] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:28.472956 10545 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: No bootstrap required, opened a new log
I20260925 11:06:28.473213 10545 ts_tablet_manager.cc:1397] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 11:06:28.473387 10543 tablet_bootstrap.cc:492] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: No bootstrap required, opened a new log
I20260925 11:06:28.473707 10543 ts_tablet_manager.cc:1397] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 11:06:28.474023 10545 raft_consensus.cc:348] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.474319 10545 raft_consensus.cc:374] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.474433 10545 raft_consensus.cc:729] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06eb478d88ba49e0a14fb307607eae4f, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.474756 10545 consensus_queue.cc:260] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.475538 10545 ts_tablet_manager.cc:1428] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:06:28.476363 10567 ts_tablet_manager.cc:1918] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:28.476588 10567 log.cc:1196] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/9b56709f328848d5b405d02233d6db06
I20260925 11:06:28.477221 10567 ts_tablet_manager.cc:1939] T 9b56709f328848d5b405d02233d6db06 P 2007ffda629b453f98850afcc1d1f081: Deleting consensus metadata
I20260925 11:06:28.475039 10543 raft_consensus.cc:348] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.478320 10543 raft_consensus.cc:374] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.479154 10023 catalog_manager.cc:4841] TS 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593): tablet 9b56709f328848d5b405d02233d6db06 (table test-table [id=46474e795df44cdd9b6ceeeabce403f5]) successfully deleted
I20260925 11:06:28.479233 10543 raft_consensus.cc:729] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2007ffda629b453f98850afcc1d1f081, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.479518 10543 consensus_queue.cc:260] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.480557 10543 ts_tablet_manager.cc:1428] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260925 11:06:28.668710 10549 raft_consensus.cc:482] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:28.668985 10549 raft_consensus.cc:504] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.670766 10549 leader_election.cc:288] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593)
I20260925 11:06:28.679149 10551 raft_consensus.cc:482] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:28.679306 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.671398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:34997 (local address 127.8.181.2:46593)
0925 11:06:28.671561 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.671567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.672937 (+ 1370us) server_negotiation.cc:407] Connection header received
0925 11:06:28.673064 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.673069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.673167 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.673259 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.673784 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.675009 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.677205 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.678110 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.678293 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.678399 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.678799 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.678961 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 11:06:28.679399 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.671284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:45289 (local address 127.8.181.1:34363)
0925 11:06:28.671531 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.671541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.672218 (+ 677us) server_negotiation.cc:407] Connection header received
0925 11:06:28.672461 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.672468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.672565 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.672656 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:28.673280 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.674213 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.676770 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.678079 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.678364 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.678484 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.678974 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.679119 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:06:28.679355 10551 raft_consensus.cc:504] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.680116 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.670909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34363 (local address 127.8.181.3:45289)
0925 11:06:28.672061 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.672088 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.672285 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.672787 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.672791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.672822 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.673125 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.673134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.674365 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.674370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.676590 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.676605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.678577 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.678582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.678795 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.678823 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.679625 (+ 802us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.679757 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":40,"thread_start_us":1640,"threads_started":1}
I20260925 11:06:28.680087 10551 leader_election.cc:288] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333)
I20260925 11:06:28.680734 10201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" is_pre_election: true
I20260925 11:06:28.680864 10475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06eb478d88ba49e0a14fb307607eae4f" is_pre_election: true
I20260925 11:06:28.680898 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" is_pre_election: true
I20260925 11:06:28.681032 10201 raft_consensus.cc:2454] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 0.
I20260925 11:06:28.681128 10475 raft_consensus.cc:2454] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 0.
I20260925 11:06:28.681267 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.671032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46593 (local address 127.8.181.3:34997)
0925 11:06:28.672820 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.672836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.672952 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.673307 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.673312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.673339 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:28.673628 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.673640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.675126 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.675150 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.677042 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.677052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.678308 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.678314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.678568 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.678639 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.678746 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.678899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"mutex_wait_us":26,"thread_start_us":670,"threads_started":1}
I20260925 11:06:28.681564 10269 leader_election.cc:302] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06eb478d88ba49e0a14fb307607eae4f, 2007ffda629b453f98850afcc1d1f081; no voters:
I20260925 11:06:28.681869 10551 raft_consensus.cc:2781] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:28.682009 10551 raft_consensus.cc:482] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:28.682089 10551 raft_consensus.cc:3037] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.682860 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2007ffda629b453f98850afcc1d1f081" is_pre_election: true
I20260925 11:06:28.684796 10551 raft_consensus.cc:504] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.685356 10340 raft_consensus.cc:2379] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06eb478d88ba49e0a14fb307607eae4f in current term 1: Already voted for candidate 2007ffda629b453f98850afcc1d1f081 in this term.
I20260925 11:06:28.685554 10551 leader_election.cc:288] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 election: Requested vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333)
I20260925 11:06:28.687038 10405 leader_election.cc:302] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06eb478d88ba49e0a14fb307607eae4f; no voters: 2007ffda629b453f98850afcc1d1f081, ff8d8154eb2c43b3869f1a66cb1c6945
I20260925 11:06:28.687114 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
I20260925 11:06:28.687350 10200 raft_consensus.cc:3037] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.687522 10549 raft_consensus.cc:3037] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.687973 10475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7442c0521f7446f8900acddf7e0b4d04" candidate_uuid: "2007ffda629b453f98850afcc1d1f081" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06eb478d88ba49e0a14fb307607eae4f"
I20260925 11:06:28.689852 10200 raft_consensus.cc:2454] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 1.
I20260925 11:06:28.690348 10269 leader_election.cc:302] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2007ffda629b453f98850afcc1d1f081, ff8d8154eb2c43b3869f1a66cb1c6945; no voters:
I20260925 11:06:28.690753 10551 raft_consensus.cc:2781] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:28.690807 10549 raft_consensus.cc:2726] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 11:06:28.690930 10551 raft_consensus.cc:686] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 LEADER]: Becoming Leader. State: Replica: 2007ffda629b453f98850afcc1d1f081, State: Running, Role: LEADER
I20260925 11:06:28.691241 10551 consensus_queue.cc:237] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.693543 10475 raft_consensus.cc:2454] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2007ffda629b453f98850afcc1d1f081 in term 1.
I20260925 11:06:28.695977 10038 catalog_manager.cc:5495] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2007ffda629b453f98850afcc1d1f081 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "2007ffda629b453f98850afcc1d1f081" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } health_report { overall_health: UNKNOWN } } }
Sep 25 11:06:28 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334386, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:28.762962 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.741096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35787 (local address 127.0.0.1:39594)
0925 11:06:28.741882 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.741906 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.742086 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.742635 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.742639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.743373 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:28.743606 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.743616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.744811 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.744817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.746109 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.746123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.747015 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.747021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.747190 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.753239 (+ 6049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:28.753272 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:28.756470 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:28.759791 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:28.759813 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:28.759823 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:28.760502 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:28.760895 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:28.760900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:28.760904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:28.761107 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:28.761654 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:28.761665 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:28.761878 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.762098 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.762462 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":1,"thread_start_us":390,"threads_started":1}
I20260925 11:06:28.762977 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.741146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.8.181.62:35787)
0925 11:06:28.742281 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.742286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.742307 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:06:28.742356 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.742360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.742417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.742617 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:28.743780 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.744648 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.746302 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.746829 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.746984 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.756751 (+ 9767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:28.756757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:28.756761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:28.756798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:28.759607 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:28.760639 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:28.760643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:28.760645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:28.760762 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:28.761225 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:28.761228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:28.761230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:28.761500 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:28.761605 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.762182 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.762673 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":271,"server-negotiator.queue_time_us":1030,"thread_start_us":879,"threads_started":1}
I20260925 11:06:28.768805 10038 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:39594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 11:06:28.769881 10038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:28.777527 10315 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c9805efc4583b3151d914c0e69c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e92f89bfaaa4ea3a74fcd3f43d4d04c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:28.778002 10454 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c9805efc4583b3151d914c0e69c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e92f89bfaaa4ea3a74fcd3f43d4d04c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:28.778204 10315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c9805efc4583b3151d914c0e69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.778069 10176 tablet_service.cc:1467] Processing CreateTablet for tablet a3e6c9805efc4583b3151d914c0e69c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e92f89bfaaa4ea3a74fcd3f43d4d04c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:28.778544 10454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c9805efc4583b3151d914c0e69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.778640 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e6c9805efc4583b3151d914c0e69c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:28.785984 10545 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: Bootstrap starting.
I20260925 11:06:28.786249 10544 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: Bootstrap starting.
I20260925 11:06:28.786410 10543 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: Bootstrap starting.
I20260925 11:06:28.787801 10545 tablet_bootstrap.cc:654] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.788007 10544 tablet_bootstrap.cc:654] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.788239 10543 tablet_bootstrap.cc:654] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:28.797909 10544 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: No bootstrap required, opened a new log
I20260925 11:06:28.798200 10544 ts_tablet_manager.cc:1397] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260925 11:06:28.801054 10543 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: No bootstrap required, opened a new log
I20260925 11:06:28.801422 10543 ts_tablet_manager.cc:1397] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260925 11:06:28.802230 10544 raft_consensus.cc:348] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.802533 10544 raft_consensus.cc:374] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.802631 10544 raft_consensus.cc:729] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8d8154eb2c43b3869f1a66cb1c6945, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.802405 10543 raft_consensus.cc:348] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.803020 10543 raft_consensus.cc:374] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.802953 10544 consensus_queue.cc:260] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.803208 10545 tablet_bootstrap.cc:492] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: No bootstrap required, opened a new log
I20260925 11:06:28.803308 10543 raft_consensus.cc:729] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2007ffda629b453f98850afcc1d1f081, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.803382 10545 ts_tablet_manager.cc:1397] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260925 11:06:28.803431 10550 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.803802 10550 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.804136 10545 raft_consensus.cc:348] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.804365 10545 raft_consensus.cc:374] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:28.804440 10545 raft_consensus.cc:729] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06eb478d88ba49e0a14fb307607eae4f, State: Initialized, Role: FOLLOWER
I20260925 11:06:28.804785 10545 consensus_queue.cc:260] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.804893 10543 consensus_queue.cc:260] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.805217 10549 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.805497 10549 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.805747 10543 ts_tablet_manager.cc:1428] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 11:06:28.806241 10551 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.807080 10551 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.807562 10545 ts_tablet_manager.cc:1428] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 11:06:28.809806 10544 ts_tablet_manager.cc:1428] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20260925 11:06:28.826054 10549 raft_consensus.cc:482] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:28.826298 10549 raft_consensus.cc:504] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.827107 10549 leader_election.cc:288] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593)
I20260925 11:06:28.827685 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c9805efc4583b3151d914c0e69c3" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" is_pre_election: true
I20260925 11:06:28.827695 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c9805efc4583b3151d914c0e69c3" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2007ffda629b453f98850afcc1d1f081" is_pre_election: true
I20260925 11:06:28.827929 10200 raft_consensus.cc:2454] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06eb478d88ba49e0a14fb307607eae4f in term 0.
I20260925 11:06:28.827966 10340 raft_consensus.cc:2454] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06eb478d88ba49e0a14fb307607eae4f in term 0.
I20260925 11:06:28.828413 10408 leader_election.cc:302] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06eb478d88ba49e0a14fb307607eae4f, ff8d8154eb2c43b3869f1a66cb1c6945; no voters:
I20260925 11:06:28.828763 10549 raft_consensus.cc:2781] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:28.828871 10549 raft_consensus.cc:482] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:28.829028 10549 raft_consensus.cc:3037] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.831562 10549 raft_consensus.cc:504] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.832715 10549 leader_election.cc:288] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 election: Requested vote from peers ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363), 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593)
I20260925 11:06:28.832800 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c9805efc4583b3151d914c0e69c3" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945"
I20260925 11:06:28.833034 10200 raft_consensus.cc:3037] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.834735 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e6c9805efc4583b3151d914c0e69c3" candidate_uuid: "06eb478d88ba49e0a14fb307607eae4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2007ffda629b453f98850afcc1d1f081"
I20260925 11:06:28.834968 10340 raft_consensus.cc:3037] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:28.836118 10200 raft_consensus.cc:2454] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06eb478d88ba49e0a14fb307607eae4f in term 1.
I20260925 11:06:28.836597 10408 leader_election.cc:302] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06eb478d88ba49e0a14fb307607eae4f, ff8d8154eb2c43b3869f1a66cb1c6945; no voters:
I20260925 11:06:28.836894 10549 raft_consensus.cc:2781] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:28.837347 10549 raft_consensus.cc:686] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [term 1 LEADER]: Becoming Leader. State: Replica: 06eb478d88ba49e0a14fb307607eae4f, State: Running, Role: LEADER
I20260925 11:06:28.837693 10340 raft_consensus.cc:2454] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06eb478d88ba49e0a14fb307607eae4f in term 1.
I20260925 11:06:28.838122 10549 consensus_queue.cc:237] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } }
I20260925 11:06:28.840200 10586 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: TxnStatusTablet state changed. Reason: New leader 06eb478d88ba49e0a14fb307607eae4f. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.840587 10586 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:28.841472 10588 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 11:06:28.841694 10588 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 11:06:28.842278 10038 catalog_manager.cc:5495] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3). New cstate: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } health_report { overall_health: HEALTHY } } }
I20260925 11:06:28.868719 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.860326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46333 (local address 127.0.0.1:33492)
0925 11:06:28.860887 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.860910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.861080 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.861592 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.861597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.861610 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:28.861816 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.861827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.863496 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.863502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.865288 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.865306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.866444 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.866450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.866685 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.866769 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:28.868085 (+ 1316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:28.868112 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.868211 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.868383 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392}
I20260925 11:06:28.868768 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.860435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.8.181.3:46333)
0925 11:06:28.861129 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.861135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.861161 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:06:28.861286 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.861292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.861396 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.861550 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:28.861994 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.863352 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.865461 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.866261 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.866402 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.867097 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:28.867943 (+ 846us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:28.868106 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.868267 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.868356 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":774,"threads_started":1}
I20260925 11:06:28.873443 10200 raft_consensus.cc:1261] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Refusing update from remote peer 06eb478d88ba49e0a14fb307607eae4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.873636 10340 raft_consensus.cc:1261] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Refusing update from remote peer 06eb478d88ba49e0a14fb307607eae4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.874131 10549 consensus_queue.cc:1035] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:28.874727 10549 consensus_queue.cc:1035] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:28.884990 10551 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: TxnStatusTablet state changed. Reason: New leader 06eb478d88ba49e0a14fb307607eae4f. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.885421 10551 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.886368 10550 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: TxnStatusTablet state changed. Reason: New leader 06eb478d88ba49e0a14fb307607eae4f. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.886731 10550 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.889601 10586 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.889835 10586 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:28.889984 10559 mvcc.cc:205] Tried to move back new op lower bound from 7333209656817979392 to 7333209656685027328. Current Snapshot: MvccSnapshot[applied={T|T < 7333209656817979392 or (T in {7333209656817979392})}]
I20260925 11:06:28.892477 10550 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.892705 10550 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P ff8d8154eb2c43b3869f1a66cb1c6945: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.909114 10549 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.909523 10549 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 06eb478d88ba49e0a14fb307607eae4f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:28.913975 10551 tablet_replica.cc:439] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06eb478d88ba49e0a14fb307607eae4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 } } peers { permanent_uuid: "2007ffda629b453f98850afcc1d1f081" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46593 } } peers { permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 } } }
I20260925 11:06:28.914278 10551 tablet_replica.cc:442] T a3e6c9805efc4583b3151d914c0e69c3 P 2007ffda629b453f98850afcc1d1f081: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:28.947759 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.941545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:52661 (local address 127.8.181.3:46333)
0925 11:06:28.941749 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:28.941756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:28.942385 (+ 629us) server_negotiation.cc:407] Connection header received
0925 11:06:28.942608 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:28.942614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:28.942702 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:28.942822 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:28.943239 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.944425 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.945847 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.946312 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.946460 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.946925 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:28.947031 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:28.947197 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:28.947333 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 11:06:28.947409 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:06:28.947844 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:28.941506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46333 (local address 127.8.181.2:52661)
0925 11:06:28.942266 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:28.942283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:28.942417 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:28.942831 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:28.942834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:28.942845 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:28.943098 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.943106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.944573 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.944577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.945700 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:28.945712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:28.946513 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:28.946520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:28.946673 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:28.946734 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:28.947196 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:28.947209 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 11:06:28.947313 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 11:06:28.947449 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":406,"threads_started":1}
I20260925 11:06:28.959283 10475 raft_consensus.cc:1261] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Refusing update from remote peer 2007ffda629b453f98850afcc1d1f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.959282 10200 raft_consensus.cc:1261] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Refusing update from remote peer 2007ffda629b453f98850afcc1d1f081: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:28.959890 10554 consensus_queue.cc:1035] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8d8154eb2c43b3869f1a66cb1c6945" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:28.960222 10551 consensus_queue.cc:1035] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06eb478d88ba49e0a14fb307607eae4f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:29.019322 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.012892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:51797 (local address 127.8.181.2:46593)
0925 11:06:29.013108 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.013117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.013763 (+ 646us) server_negotiation.cc:407] Connection header received
0925 11:06:29.013983 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.013988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.014074 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.014173 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:29.014757 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.015721 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.017357 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.017908 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.018068 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.018519 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:29.018684 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:29.018779 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.018953 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.019033 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 11:06:29.019488 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.012827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46593 (local address 127.8.181.3:51797)
0925 11:06:29.013659 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:29.013674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:29.013833 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:29.014188 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:29.014192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:29.014204 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:29.014575 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:29.014587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.015863 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.015869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:29.017215 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:29.017228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.018102 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.018109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:29.018253 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.018311 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:29.018800 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:29.018818 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 11:06:29.018911 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 11:06:29.019088 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":464,"threads_started":1}
I20260925 11:06:29.055038 10037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:29.055325 10037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:29.058491 10037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22a82e879e33494fb100c8df3f20d0dc: Sending DeleteTablet for 3 replicas of tablet 7442c0521f7446f8900acddf7e0b4d04
I20260925 11:06:29.059276 10315 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:29 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:37424
I20260925 11:06:29.059420 10176 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:29 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:42238
I20260925 11:06:29.060122 10454 tablet_service.cc:1514] Processing DeleteTablet for tablet 7442c0521f7446f8900acddf7e0b4d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:29 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:33490
I20260925 11:06:29.074688 10605 tablet_replica.cc:330] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: stopping tablet replica
I20260925 11:06:29.074931 10605 raft_consensus.cc:2229] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:29.075453 10605 raft_consensus.cc:2258] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:29.077286 10605 ts_tablet_manager.cc:1905] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:29.085075 10607 tablet_replica.cc:330] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: stopping tablet replica
I20260925 11:06:29.085520 10607 raft_consensus.cc:2229] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:29.085911 10607 raft_consensus.cc:2258] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:29.086457 10606 tablet_replica.cc:330] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: stopping tablet replica
I20260925 11:06:29.087836 10607 ts_tablet_manager.cc:1905] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:29.088171 10605 ts_tablet_manager.cc:1918] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:29.088178 10606 raft_consensus.cc:2229] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:29.088318 10605 log.cc:1196] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/7442c0521f7446f8900acddf7e0b4d04
I20260925 11:06:29.088589 10606 raft_consensus.cc:2258] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:29.088820 10605 ts_tablet_manager.cc:1939] T 7442c0521f7446f8900acddf7e0b4d04 P 2007ffda629b453f98850afcc1d1f081: Deleting consensus metadata
I20260925 11:06:29.089792 10606 ts_tablet_manager.cc:1905] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:29.089977 10023 catalog_manager.cc:4841] TS 2007ffda629b453f98850afcc1d1f081 (127.8.181.2:46593): tablet 7442c0521f7446f8900acddf7e0b4d04 (table test-table [id=6527f1c8b5a347a8863bd4ec0118a617]) successfully deleted
I20260925 11:06:29.097446 10607 ts_tablet_manager.cc:1918] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:29.097596 10606 ts_tablet_manager.cc:1918] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:29.097730 10606 log.cc:1196] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/7442c0521f7446f8900acddf7e0b4d04
I20260925 11:06:29.097667 10607 log.cc:1196] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/7442c0521f7446f8900acddf7e0b4d04
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:29.098268 10606 ts_tablet_manager.cc:1939] T 7442c0521f7446f8900acddf7e0b4d04 P ff8d8154eb2c43b3869f1a66cb1c6945: Deleting consensus metadata
I20260925 11:06:29.098304 10607 ts_tablet_manager.cc:1939] T 7442c0521f7446f8900acddf7e0b4d04 P 06eb478d88ba49e0a14fb307607eae4f: Deleting consensus metadata
I20260925 11:06:29.099457 10026 catalog_manager.cc:4841] TS ff8d8154eb2c43b3869f1a66cb1c6945 (127.8.181.1:34363): tablet 7442c0521f7446f8900acddf7e0b4d04 (table test-table [id=6527f1c8b5a347a8863bd4ec0118a617]) successfully deleted
I20260925 11:06:29.099745 10026 catalog_manager.cc:4841] TS 06eb478d88ba49e0a14fb307607eae4f (127.8.181.3:46333): tablet 7442c0521f7446f8900acddf7e0b4d04 (table test-table [id=6527f1c8b5a347a8863bd4ec0118a617]) successfully deleted
I20260925 11:06:29.105090 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.074720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.8.181.1:34363)
0925 11:06:29.074930 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.074936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.087160 (+ 12224us) server_negotiation.cc:407] Connection header received
0925 11:06:29.087441 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.087446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.087528 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.087621 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:29.089765 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.090856 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.092772 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.093419 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.093511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.099209 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:29.099226 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:29.099248 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:29.099310 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:29.102303 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.103045 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.103054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.103058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.103198 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.103587 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.103592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.103595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.103933 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:29.104247 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.104530 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.104752 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 11:06:29.105885 10241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42248
I20260925 11:06:29.123237 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.109016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.8.181.62:35787)
0925 11:06:29.109267 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.109273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.109436 (+ 163us) server_negotiation.cc:407] Connection header received
0925 11:06:29.109554 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.109560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.109649 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.109759 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:29.111059 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.112322 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.113704 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.114382 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.114568 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.116866 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:29.116873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:29.116877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:29.116942 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:29.120483 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.121192 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.121198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.121202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.121346 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.121683 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.121688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.121692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.122050 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:29.122323 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.122639 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.122886 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260925 11:06:29.123835 10037 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39596
I20260925 11:06:29.125730 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10110
I20260925 11:06:29.139410 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.126641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.8.181.1:34363)
0925 11:06:29.126834 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.126840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.127293 (+ 453us) server_negotiation.cc:407] Connection header received
0925 11:06:29.127479 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.127484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.127570 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.127678 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:29.128782 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.129508 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.130900 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.131475 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.131646 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.133942 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:29.133949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:29.133953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:29.133994 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:29.136600 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.137249 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.137255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.137258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.137386 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.137744 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.137749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.137753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.138066 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:29.138384 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.138786 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.139032 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 11:06:29.139997 10241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42256
W20260925 11:06:29.140524 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10110: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:29.140712 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10110
I20260925 11:06:29.158453 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10250
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:29.176873 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.159318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.8.181.2:46593)
0925 11:06:29.159501 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.159505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.159545 (+ 40us) server_negotiation.cc:407] Connection header received
0925 11:06:29.159924 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.159928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.159991 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.160084 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:29.161621 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.162304 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.164103 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.164558 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.164690 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.169773 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:29.169796 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:29.169813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:29.169866 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:29.173651 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.174451 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.174462 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.174489 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.174658 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.175109 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.175114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.175118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.175553 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:29.175893 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.176328 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.176565 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 11:06:29.177560 10380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450
W20260925 11:06:29.178449 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:29.178946 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10250
I20260925 11:06:29.201521 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10389
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[9994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:29.219806 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:29.202615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.8.181.3:46333)
0925 11:06:29.202782 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:29.202786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:29.202802 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:06:29.202885 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:29.202888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:29.202972 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:29.203072 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:29.204721 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.205511 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.207447 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:29.207864 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:29.207965 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:29.213187 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:29.213216 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:29.213230 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:29.213268 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:29.216733 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.217479 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.217490 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.217495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.217656 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:29.218151 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:29.218156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:29.218159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:29.218574 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:29.218913 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:29.219233 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 11:06:29.219464 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 11:06:29.227339 10519 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33494
W20260925 11:06:29.228084 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:29.228266 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10389
I20260925 11:06:29.248544 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10007
I20260925 11:06:29.249707 10079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:29.366110 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10007
2026-09-25T11:06:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4874 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[10616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[10616](info): set up 2 sockets
Sep 25 11:06:29 dist-test-slave-6042 krb5kdc[10616](info): commencing operation
krb5kdc: starting...
W20260925 11:06:30.554944 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:30 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:30.755502 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32837
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:32837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:31.079320 10629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:31.079646 10629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:31.079737 10629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:31.087407 10629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:31.087529 10629 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 11:06:31.087570 10629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:31.087631 10629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:31.087673 10629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:31.087736 10629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:31.097983 10629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:32837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:31.099314 10629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:31.100584 10629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:31.112717 10635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:31.112732 10637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:31.112844 10634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:31.113575 10629 server_base.cc:1034] running on GCE node
I20260925 11:06:31.114418 10629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:31.116029 10629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:31.117316 10629 hybrid_clock.cc:648] HybridClock initialized: now 1790334391117261 us; error 34 us; skew 500 ppm
I20260925 11:06:31.117698 10629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:31 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334391, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:31.120982 10629 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:31.122592 10629 webserver.cc:460] Webserver started at http://127.8.181.62:34549/ using document root <none> and password file <none>
I20260925 11:06:31.123355 10629 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:31.123492 10629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:31.123811 10629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:31.127004 10629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "89c1f4ef1c434f62a5830cfb0fac3470"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "cbcbbec1756b5cb56ed7a14476586790"
server_key_iv: "bfd70c6f4b6e2a413d76c923c07b7742"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.127828 10629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "89c1f4ef1c434f62a5830cfb0fac3470"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "cbcbbec1756b5cb56ed7a14476586790"
server_key_iv: "bfd70c6f4b6e2a413d76c923c07b7742"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.133193 10629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:31.136823 10644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:31.138213 10629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 11:06:31.138409 10629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "89c1f4ef1c434f62a5830cfb0fac3470"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "cbcbbec1756b5cb56ed7a14476586790"
server_key_iv: "bfd70c6f4b6e2a413d76c923c07b7742"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.138677 10629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:31.157104 10629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:31.161652 10629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:31.161968 10629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:31.183673 10629 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:32837
I20260925 11:06:31.183693 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:32837 every 8 connection(s)
I20260925 11:06:31.185104 10629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:31.188902 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:31.194796 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10629
I20260925 11:06:31.195017 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:31.195672 8916 external_mini_cluster.cc:1428] Setting key e1e194eb5f41769f44fd8b6e5c724dba
I20260925 11:06:31.200387 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Bootstrap starting.
I20260925 11:06:31.204176 10706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:31.205425 10706 log.cc:824] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:31.208485 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: No bootstrap required, opened a new log
Sep 25 11:06:31 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:31.217990 10706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } }
I20260925 11:06:31.218485 10706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:31.218641 10706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c1f4ef1c434f62a5830cfb0fac3470, State: Initialized, Role: FOLLOWER
I20260925 11:06:31.219288 10706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } }
I20260925 11:06:31.219599 10706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:31.219759 10706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:31.219946 10706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:31.220026 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:31.197857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.8.181.62:32837)
0925 11:06:31.198743 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:31.198755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:31.198797 (+ 42us) server_negotiation.cc:407] Connection header received
0925 11:06:31.200501 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:31.200533 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:31.200976 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:31.201299 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:31.203211 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.204565 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.206401 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.207126 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.207317 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:31.211847 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:31.211858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:31.211876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:31.211918 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:31.215269 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.216197 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.216205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.216215 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.216388 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.216787 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.216800 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.216802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.217391 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:31.217605 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:31.218111 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 11:06:31.218683 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":422,"threads_started":1}
I20260925 11:06:31.222853 10706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } }
I20260925 11:06:31.223417 10706 leader_election.cc:302] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c1f4ef1c434f62a5830cfb0fac3470; no voters:
I20260925 11:06:31.223934 10706 leader_election.cc:288] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:31.224094 10711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:31.224895 10711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [term 1 LEADER]: Becoming Leader. State: Replica: 89c1f4ef1c434f62a5830cfb0fac3470, State: Running, Role: LEADER
I20260925 11:06:31.225441 10711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } }
I20260925 11:06:31.226051 10706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:31.228124 10713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c1f4ef1c434f62a5830cfb0fac3470. Latest consensus state: current_term: 1 leader_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } } }
I20260925 11:06:31.228510 10713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:31.231251 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c1f4ef1c434f62a5830cfb0fac3470" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32837 } } }
I20260925 11:06:31.232620 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:31.232847 10718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:31.237577 10718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:31.247094 10718 catalog_manager.cc:1347] Generated new cluster ID: 13d81340757e434ba03a3592149426a8
I20260925 11:06:31.247231 10718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:31.257481 10718 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:31.258924 10718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:31.267899 10718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Generated new TSK 0
I20260925 11:06:31.268816 10718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:31.320289 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32837
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:31.652894 10732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:31.653227 10732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:31.653342 10732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:31.660760 10732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:31.660881 10732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:31.660969 10732 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 11:06:31.661022 10732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:31.661089 10732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:31.671036 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:31.672428 10732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:31.673768 10732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:31.687482 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:31.687557 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:31.687587 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:31.688539 10732 server_base.cc:1034] running on GCE node
I20260925 11:06:31.689172 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:31.690222 10732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:31.691429 10732 hybrid_clock.cc:648] HybridClock initialized: now 1790334391691370 us; error 61 us; skew 500 ppm
I20260925 11:06:31.691763 10732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:31 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334391, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:31.694924 10732 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:31.696725 10732 webserver.cc:460] Webserver started at http://127.8.181.1:40447/ using document root <none> and password file <none>
I20260925 11:06:31.697479 10732 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:31.697649 10732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:31.697984 10732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:31.701064 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "b14669471e0d391312a74ae6804c7412"
server_key_iv: "c63ef8e71e357dfac77cfa446277366a"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.701880 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "b14669471e0d391312a74ae6804c7412"
server_key_iv: "c63ef8e71e357dfac77cfa446277366a"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.707528 10732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 11:06:31.711359 10747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:31.712805 10732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:06:31.712945 10732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
format_stamp: "Formatted at 2026-09-25 11:06:31 on dist-test-slave-6042"
server_key: "b14669471e0d391312a74ae6804c7412"
server_key_iv: "c63ef8e71e357dfac77cfa446277366a"
server_key_version: "encryptionkey@0"
I20260925 11:06:31.713124 10732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:31.744156 10732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:31.749250 10732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:31.749531 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:31.752379 10732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:31.752548 10732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:31.752694 10732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:31.752805 10732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:31 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334391, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:31.778043 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:31.753018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:58889 (local address 127.8.181.62:32837)
0925 11:06:31.753767 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:31.753774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:31.754894 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 11:06:31.756946 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:31.756953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:31.757066 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:31.757233 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:31.759553 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.761019 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.763645 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.764407 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.764573 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:31.770513 (+ 5940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:31.770524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:31.770530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:31.770576 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:31.773875 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.774954 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.774960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.774964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.775141 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.775746 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.775752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.775756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.776083 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:31.776267 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:31.777397 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0925 11:06:31.777659 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":501,"threads_started":1}
I20260925 11:06:31.779752 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:31.753755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.1:58889)
0925 11:06:31.754705 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:31.754752 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:31.756643 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:31.757523 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:31.757539 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:31.758330 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:31.759325 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:31.759345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.761250 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.761261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:31.763503 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:31.763528 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.764688 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.764698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:31.765040 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:31.765888 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:31.765931 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:31.770115 (+ 4184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:31.774041 (+ 3926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:31.774051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:31.774071 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:31.774815 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:31.775306 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:31.775311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:31.775315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:31.775545 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:31.776365 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:31.776379 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:31.776913 (+ 534us) client_negotiation.cc:769] Sending connection context
0925 11:06:31.777312 (+ 399us) client_negotiation.cc:240] Negotiation successful
0925 11:06:31.777954 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":416,"threads_started":1}
I20260925 11:06:31.794757 10732 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:39165
I20260925 11:06:31.794791 10867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:39165 every 8 connection(s)
I20260925 11:06:31.796514 10732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:31.799062 10868 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32837
I20260925 11:06:31.799335 10868 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:31.800253 10868 heartbeater.cc:511] Master 127.8.181.62:32837 requested a full tablet report, sending...
I20260925 11:06:31.800280 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10732
I20260925 11:06:31.800637 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:31.801427 8916 external_mini_cluster.cc:1428] Setting key 9b6c436d34271339388d60ccaa665e38
I20260925 11:06:31.803241 10661 ts_manager.cc:194] Registered new tserver with Master: 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165)
I20260925 11:06:31.805454 10661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:58889
I20260925 11:06:31.805792 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:31.787838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:44679 (local address 127.8.181.62:32837)
0925 11:06:31.788070 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:31.788078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:31.789112 (+ 1034us) server_negotiation.cc:407] Connection header received
0925 11:06:31.789252 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:31.789261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:31.789357 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:31.789471 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:31.791418 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.792774 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.795017 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.795669 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.795777 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:31.798551 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:31.798558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:31.798563 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:31.798608 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:31.802261 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.803240 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.803246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.803250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.803426 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:31.803919 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:31.803924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:31.803927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:31.804306 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:31.804463 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:31.805002 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 11:06:31.805240 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:06:31.806243 10852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:31.787930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.1:44679)
0925 11:06:31.788977 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:31.788998 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:31.789140 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:31.789725 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:31.789736 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:31.790573 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:31.791187 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:31.791207 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.792929 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.792934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:31.794836 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:31.794852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:31.796099 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:31.796107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:31.796287 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:31.796994 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:31.797020 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:31.798327 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:31.802434 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:31.802438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:31.802441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:31.803070 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:31.803538 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:31.803543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:31.803547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:31.803722 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:31.804530 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:31.804536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:31.804727 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 11:06:31.805490 (+ 763us) client_negotiation.cc:240] Negotiation successful
0925 11:06:31.805828 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":755,"threads_started":1}
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:31.846767 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32837
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:32.175864 10872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:32.176221 10872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:32.176405 10872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:32.184475 10872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:32.184605 10872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:32.184738 10872 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 11:06:32.184823 10872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:32.184975 10872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:32.200733 10872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:32.202334 10872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:32.203949 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:32.218007 10877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:32.218008 10878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:32.218137 10880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:32.219178 10872 server_base.cc:1034] running on GCE node
I20260925 11:06:32.219827 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:32.220845 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:32.222075 10872 hybrid_clock.cc:648] HybridClock initialized: now 1790334392222015 us; error 55 us; skew 500 ppm
I20260925 11:06:32.222397 10872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:32.225426 10872 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:32.227025 10872 webserver.cc:460] Webserver started at http://127.8.181.2:41121/ using document root <none> and password file <none>
I20260925 11:06:32.227808 10872 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:32.227959 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:32.228284 10872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:32.231302 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "e49cf198c46c48048ba9d00b04c7cf6b"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "ecdf076c33086e71ddd7fba3391020ed"
server_key_iv: "a739c6a35903614c2c36e5ca315475cd"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.232103 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "e49cf198c46c48048ba9d00b04c7cf6b"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "ecdf076c33086e71ddd7fba3391020ed"
server_key_iv: "a739c6a35903614c2c36e5ca315475cd"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.237373 10872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:32.241003 10887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:32.242580 10872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 11:06:32.242816 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "e49cf198c46c48048ba9d00b04c7cf6b"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "ecdf076c33086e71ddd7fba3391020ed"
server_key_iv: "a739c6a35903614c2c36e5ca315475cd"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.243048 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:32.266743 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:32.271272 10872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:32.271585 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:32.274405 10872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:32.274580 10872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 11:06:32.274729 10872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:32.274816 10872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:32.300323 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.275380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:40215 (local address 127.8.181.62:32837)
0925 11:06:32.275561 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.275569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.277514 (+ 1945us) server_negotiation.cc:407] Connection header received
0925 11:06:32.279629 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.279639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.279740 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.279862 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:32.282094 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.283058 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.285723 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.286171 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.286313 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.291366 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:32.291371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:32.291397 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:32.291453 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:32.295707 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.296865 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.296869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.296871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.296996 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.297751 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.297758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.297761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.298149 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:32.298314 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.299305 (+ 991us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.299798 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 11:06:32.300896 10896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.276492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.2:40215)
0925 11:06:32.277337 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.277384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.279280 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.280126 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.280140 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.280943 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:32.281811 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.281847 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.283258 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.283267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.285567 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.285579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.286414 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.286421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.286692 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.287748 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:32.287788 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:32.291101 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:32.295909 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.295920 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.295944 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.296670 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.297127 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.297132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.297136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.297582 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.298390 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:32.298396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:32.298841 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.299184 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.299676 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":397,"threads_started":1}
I20260925 11:06:32.325707 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.307148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:39487 (local address 127.8.181.62:32837)
0925 11:06:32.307391 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.307400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.308004 (+ 604us) server_negotiation.cc:407] Connection header received
0925 11:06:32.308218 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.308226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.308331 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.308468 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:32.309469 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.310826 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.313636 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.314457 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.314646 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.317629 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:32.317640 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:32.317649 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:32.317705 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:32.321841 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.322927 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.322934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.322938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.323173 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.323719 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.323723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.323725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.324255 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:32.324409 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.325028 (+ 619us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.325266 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 11:06:32.325801 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.307203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.2:39487)
0925 11:06:32.307874 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.307904 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.308070 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.308479 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.308482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.308984 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:32.309315 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.309325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.311030 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.311035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.313388 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.313407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.314937 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.314946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.315200 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.316052 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:32.316074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:32.317382 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:32.322040 (+ 4658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.322048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.322057 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.322787 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.323345 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.323354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.323361 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.323573 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.324479 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:32.324482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:32.324737 (+ 255us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.324937 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.325315 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":382,"threads_started":1}
I20260925 11:06:32.333859 10872 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:45249
I20260925 11:06:32.333901 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:45249 every 8 connection(s)
I20260925 11:06:32.335150 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:32.336073 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 10872
I20260925 11:06:32.336242 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:32.336691 8916 external_mini_cluster.cc:1428] Setting key c6f52d461922445bf7fdd189133a0ac7
I20260925 11:06:32.337431 11007 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32837
I20260925 11:06:32.337689 11007 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:32.338363 11007 heartbeater.cc:511] Master 127.8.181.62:32837 requested a full tablet report, sending...
I20260925 11:06:32.339838 10661 ts_manager.cc:194] Registered new tserver with Master: e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249)
I20260925 11:06:32.341136 10661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:40215
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:32.382117 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32837
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:32.713780 11011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:32.714044 11011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:32.714143 11011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:32.721536 11011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:32.721652 11011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:32.721737 11011 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 11:06:32.721788 11011 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:06:32.721855 11011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:32.731932 11011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:32837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:32.733188 11011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:32.734524 11011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:32.753710 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:32.753863 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:32.755476 11019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:32.760226 11011 server_base.cc:1034] running on GCE node
I20260925 11:06:32.761209 11011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:32.762563 11011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:32.763805 11011 hybrid_clock.cc:648] HybridClock initialized: now 1790334392763736 us; error 62 us; skew 500 ppm
I20260925 11:06:32.764196 11011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:32.768385 11011 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:32.770620 11011 webserver.cc:460] Webserver started at http://127.8.181.3:37621/ using document root <none> and password file <none>
I20260925 11:06:32.771659 11011 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:32.771816 11011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:32.772228 11011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:32.776859 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "85d00140c61b46aaabe57cb156aeb0f5"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "9570e0991bc72fddef3b3674c9fcfd3e"
server_key_iv: "679f66b56e9f90874ba47782b24c2cdb"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.777958 11011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "85d00140c61b46aaabe57cb156aeb0f5"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "9570e0991bc72fddef3b3674c9fcfd3e"
server_key_iv: "679f66b56e9f90874ba47782b24c2cdb"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.785923 11011 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260925 11:06:32.790596 11026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:32.792315 11011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:06:32.792491 11011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "85d00140c61b46aaabe57cb156aeb0f5"
format_stamp: "Formatted at 2026-09-25 11:06:32 on dist-test-slave-6042"
server_key: "9570e0991bc72fddef3b3674c9fcfd3e"
server_key_iv: "679f66b56e9f90874ba47782b24c2cdb"
server_key_version: "encryptionkey@0"
I20260925 11:06:32.792718 11011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:32.808878 10868 heartbeater.cc:503] Master 127.8.181.62:32837 was elected leader, sending a full tablet report...
I20260925 11:06:32.812109 11011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:32.816754 11011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:32.817067 11011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:32.820166 11011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:32.820290 11011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:32.820472 11011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:32.820580 11011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:32.845937 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.820624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55039 (local address 127.8.181.62:32837)
0925 11:06:32.820874 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.820883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.822355 (+ 1472us) server_negotiation.cc:407] Connection header received
0925 11:06:32.824609 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.824615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.824719 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.824839 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:32.827722 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.828708 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.831998 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.832647 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.832804 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.837350 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:32.837360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:32.837366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:32.837422 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:32.840921 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.842325 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.842335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.842341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.842578 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.843307 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.843314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.843319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.843646 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:32.843766 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.845200 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.845505 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:06:32.848093 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.821364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.3:55039)
0925 11:06:32.822164 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.822214 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.824300 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.825181 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.825194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.826178 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:32.827410 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.827450 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.828917 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.828924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.831779 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.831804 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.832925 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.832938 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.833174 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.833970 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:32.833992 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:32.837061 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:32.841269 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.841280 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.841307 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.842115 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.842760 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.842769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.842775 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.843139 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.843792 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:32.843807 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:32.844445 (+ 638us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.845003 (+ 558us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.845806 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":354,"threads_started":1}
I20260925 11:06:32.868827 11011 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:46663
I20260925 11:06:32.868866 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:46663 every 8 connection(s)
I20260925 11:06:32.870200 11011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:32.872462 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11011
I20260925 11:06:32.872653 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:32.872816 11146 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32837
I20260925 11:06:32.873040 11146 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:32.873220 8916 external_mini_cluster.cc:1428] Setting key bf5acab331ed05f7c5111c5ee3d6d714
I20260925 11:06:32.873857 11146 heartbeater.cc:511] Master 127.8.181.62:32837 requested a full tablet report, sending...
I20260925 11:06:32.875380 10661 ts_manager.cc:194] Registered new tserver with Master: 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:32.876453 10661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:55039
I20260925 11:06:32.876528 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:06:32.876991 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.858439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.8.181.3:59265)
0925 11:06:32.863174 (+ 4735us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.863197 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.863408 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.863896 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.863900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.864428 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:32.864916 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.864930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.865939 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.865947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.867380 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.867393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.868356 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.868362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.868536 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.869230 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:32.869247 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:32.870003 (+ 756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:32.873978 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.873986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.873990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.874536 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.874962 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:32.874966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:32.874970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:32.875119 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:32.875872 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:32.875877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:32.876059 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.876190 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.876485 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4494,"thread_start_us":4330,"threads_started":1}
I20260925 11:06:32.877879 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.859029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:59265 (local address 127.8.181.62:32837)
0925 11:06:32.859244 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.859253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.863349 (+ 4096us) server_negotiation.cc:407] Connection header received
0925 11:06:32.863582 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.863591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.863723 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.863879 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:32.865079 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.865795 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.867529 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.868148 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.868303 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.870222 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:32.870227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:32.870230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:32.870288 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:32.873811 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.874663 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.874670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.874676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.874842 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.875273 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.875281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.875286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.875657 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:32.875803 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.877294 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.877531 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 11:06:32 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:32.925668 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.911521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.8.181.62:32837)
0925 11:06:32.911811 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.911819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.912033 (+ 214us) server_negotiation.cc:407] Connection header received
0925 11:06:32.912191 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.912194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.912260 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.912348 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:32.913701 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.914426 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.915694 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.916100 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.916193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.920311 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:32.920316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:32.920319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:32.920354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:32.922974 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.923684 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.923690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.923694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.923815 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:32.924199 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:32.924204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:32.924208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:32.924486 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:32.924775 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.925045 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.925272 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260925 11:06:32.929945 10661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:32.937045 10661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:32.972975 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.952932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:39165 (local address 127.0.0.1:41058)
0925 11:06:32.964113 (+ 11181us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.964145 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.964259 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.964976 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.964985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.965013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:32.965423 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.965436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.967511 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.967516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.969894 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.969920 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.971865 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.971871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.972070 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.972139 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.972268 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.972477 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11055,"mutex_wait_us":67,"thread_start_us":10894,"threads_started":1}
I20260925 11:06:32.973256 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.950779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.8.181.2:45249)
0925 11:06:32.965608 (+ 14829us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.965616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.965640 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:06:32.965733 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.965741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.965963 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.966170 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:32.966841 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.968297 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.970802 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.972111 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.972265 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.972450 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.972752 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.972943 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14654,"thread_start_us":14513,"threads_started":1}
I20260925 11:06:32.973811 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.949989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46663 (local address 127.0.0.1:53238)
0925 11:06:32.962384 (+ 12395us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.962439 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.962685 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.966739 (+ 4054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.966745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.966774 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:32.967180 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.967192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.969110 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.969118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.971152 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.971167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.973113 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.973119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.973297 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.973324 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.973410 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.973545 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12092,"mutex_wait_us":1,"thread_start_us":12892,"threads_started":1}
I20260925 11:06:32.973857 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.953554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.8.181.1:39165)
0925 11:06:32.963630 (+ 10076us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.963642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.964385 (+ 743us) server_negotiation.cc:407] Connection header received
0925 11:06:32.964499 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.964507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.964748 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.964967 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:32.965623 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.967346 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.970092 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.971668 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.971803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.972015 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.973496 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.973653 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9924,"thread_start_us":9783,"threads_started":1}
I20260925 11:06:32.974404 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.950246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.8.181.3:46663)
0925 11:06:32.966131 (+ 15885us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:32.966140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:32.966170 (+ 30us) server_negotiation.cc:407] Connection header received
0925 11:06:32.966263 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:32.966272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:32.966476 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:32.967377 (+ 901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:32.967664 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.968968 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.971337 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.972711 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.972827 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.973035 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:32.974071 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0925 11:06:32.974188 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15712,"thread_start_us":15520,"threads_started":1}
I20260925 11:06:32.974897 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:32.950124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.0.0.1:56912)
0925 11:06:32.963146 (+ 13022us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:32.963173 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:32.963342 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:32.966185 (+ 2843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:32.966191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:32.966217 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:32.966639 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.966653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.968452 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.968459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.970601 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:32.970616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:32.972339 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:32.972346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:32.972529 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:32.972563 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:06:32.972663 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 11:06:32.972814 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12797,"mutex_wait_us":1,"thread_start_us":11807,"threads_started":1}
I20260925 11:06:32.978235 11076 tablet_service.cc:1467] Processing CreateTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 (DEFAULT_TABLE table=test-table [id=968e5c581d5049e5b96206b8282290f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:32.978235 10798 tablet_service.cc:1467] Processing CreateTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 (DEFAULT_TABLE table=test-table [id=968e5c581d5049e5b96206b8282290f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:32.979943 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9858dfd957e46b8beca0c2c88e134e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:32.979971 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9858dfd957e46b8beca0c2c88e134e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:32.981913 10937 tablet_service.cc:1467] Processing CreateTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 (DEFAULT_TABLE table=test-table [id=968e5c581d5049e5b96206b8282290f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:32.983167 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9858dfd957e46b8beca0c2c88e134e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:32.995497 11165 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Bootstrap starting.
I20260925 11:06:32.997848 11167 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Bootstrap starting.
I20260925 11:06:32.998639 11166 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Bootstrap starting.
I20260925 11:06:32.999245 11165 tablet_bootstrap.cc:654] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.000573 11165 log.cc:824] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:33.002053 11166 tablet_bootstrap.cc:654] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.002161 11167 tablet_bootstrap.cc:654] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.003381 11167 log.cc:824] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:33.003407 11166 log.cc:824] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:33.004724 11165 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: No bootstrap required, opened a new log
I20260925 11:06:33.005033 11165 ts_tablet_manager.cc:1397] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 11:06:33.006155 11166 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: No bootstrap required, opened a new log
I20260925 11:06:33.006410 11166 ts_tablet_manager.cc:1397] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 11:06:33.006415 11167 tablet_bootstrap.cc:492] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: No bootstrap required, opened a new log
I20260925 11:06:33.006709 11167 ts_tablet_manager.cc:1397] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260925 11:06:33.013887 11166 raft_consensus.cc:348] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.014341 11166 raft_consensus.cc:374] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.014436 11166 raft_consensus.cc:729] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff9f32a7404dcb9adaf0214ece0d3c, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.015189 11166 consensus_queue.cc:260] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.016422 11165 raft_consensus.cc:348] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.016815 11166 ts_tablet_manager.cc:1428] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 11:06:33.016851 11165 raft_consensus.cc:374] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.016963 11165 raft_consensus.cc:729] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.017634 11165 consensus_queue.cc:260] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.018231 11167 raft_consensus.cc:348] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.018703 11167 raft_consensus.cc:374] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.018821 11167 raft_consensus.cc:729] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.019420 11167 consensus_queue.cc:260] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.020500 11146 heartbeater.cc:503] Master 127.8.181.62:32837 was elected leader, sending a full tablet report...
I20260925 11:06:33.021236 11167 ts_tablet_manager.cc:1428] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260925 11:06:33.022979 11007 heartbeater.cc:503] Master 127.8.181.62:32837 was elected leader, sending a full tablet report...
I20260925 11:06:33.023720 11165 ts_tablet_manager.cc:1428] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
W20260925 11:06:33.049335 10869 tablet.cc:2369] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:33.087355 11008 tablet.cc:2369] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:33.122134 11147 tablet.cc:2369] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:33.176158 11173 raft_consensus.cc:482] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:33.176522 11173 raft_consensus.cc:504] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.178350 11173 leader_election.cc:288] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249), 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165)
I20260925 11:06:33.185685 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.178846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:49845 (local address 127.8.181.2:45249)
0925 11:06:33.179299 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.179305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.179501 (+ 196us) server_negotiation.cc:407] Connection header received
0925 11:06:33.179708 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.179712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.179767 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.179833 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:33.180309 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.181396 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.183755 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.184629 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.184786 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.184893 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.185256 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.185394 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260925 11:06:33.185736 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.178681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.8.181.3:49845)
0925 11:06:33.179392 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.179406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.179548 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.179882 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.179886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.179913 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:33.180172 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.180178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.181523 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.181528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.183542 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.183554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.184839 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.184845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.185039 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.185071 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.185200 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.185359 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":425,"threads_started":1}
I20260925 11:06:33.186223 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.178901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52475 (local address 127.8.181.1:39165)
0925 11:06:33.179104 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.179110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.179130 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:06:33.179321 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.179328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.179413 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.179532 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:33.179947 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.181116 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.183703 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.184911 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.185038 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.185131 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.185826 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.185968 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 11:06:33.185757 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.178657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:39165 (local address 127.8.181.3:52475)
0925 11:06:33.178936 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.178961 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.179101 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.179568 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.179576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.179602 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:33.179777 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.179785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.181267 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.181270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.183492 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.183513 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.185061 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.185066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.185293 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.185321 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.185394 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.185516 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2}
I20260925 11:06:33.186960 10962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9858dfd957e46b8beca0c2c88e134e4" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" is_pre_election: true
I20260925 11:06:33.187500 10962 raft_consensus.cc:2454] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 0.
I20260925 11:06:33.187579 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9858dfd957e46b8beca0c2c88e134e4" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" is_pre_election: true
I20260925 11:06:33.188119 10818 raft_consensus.cc:2454] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 0.
I20260925 11:06:33.188187 11029 leader_election.cc:302] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:33.188566 11173 raft_consensus.cc:2781] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:33.188742 11173 raft_consensus.cc:482] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:33.188875 11173 raft_consensus.cc:3037] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.191150 11173 raft_consensus.cc:504] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.191874 11173 leader_election.cc:288] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 election: Requested vote from peers e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249), 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165)
I20260925 11:06:33.192381 10962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9858dfd957e46b8beca0c2c88e134e4" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49cf198c46c48048ba9d00b04c7cf6b"
I20260925 11:06:33.192418 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9858dfd957e46b8beca0c2c88e134e4" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
I20260925 11:06:33.192607 10818 raft_consensus.cc:3037] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.192618 10962 raft_consensus.cc:3037] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.195887 10962 raft_consensus.cc:2454] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 1.
I20260925 11:06:33.195887 10818 raft_consensus.cc:2454] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 1.
I20260925 11:06:33.196369 11029 leader_election.cc:302] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:33.196712 11173 raft_consensus.cc:2781] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:33.197222 11173 raft_consensus.cc:686] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 LEADER]: Becoming Leader. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Running, Role: LEADER
I20260925 11:06:33.197777 11173 consensus_queue.cc:237] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } }
I20260925 11:06:33.202610 10661 catalog_manager.cc:5495] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3). New cstate: current_term: 1 leader_uuid: "85d00140c61b46aaabe57cb156aeb0f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } health_report { overall_health: UNKNOWN } } }
Sep 25 11:06:33 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334391, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:33.294744 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.281184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.8.181.62:32837)
0925 11:06:33.281373 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.281377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.281974 (+ 597us) server_negotiation.cc:407] Connection header received
0925 11:06:33.282155 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.282159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.282219 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.282300 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:33.283631 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.284366 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.285691 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.286152 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.286283 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.289579 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:33.289590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:33.289594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:33.289625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:33.292074 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:33.292758 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:33.292764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:33.292767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:33.292887 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:33.293241 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:33.293248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:33.293252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:33.293517 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:33.293668 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.294101 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.294421 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":73}
I20260925 11:06:33.294778 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.281195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32837 (local address 127.0.0.1:52104)
0925 11:06:33.281862 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.281878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.282046 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.282362 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.282366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.283221 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:33.283462 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.283493 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.284489 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.284495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.285477 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.285488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.286324 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.286330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.286551 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.287221 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:33.287241 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:33.289384 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:33.292205 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:33.292218 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:33.292226 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:33.292601 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:33.292987 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:33.292993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:33.292997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:33.293134 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:33.293647 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:33.293655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:33.293830 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.293989 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.294297 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":415,"threads_started":1}
I20260925 11:06:33.300036 10661 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:52104:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 11:06:33.301061 10661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:33.308125 10798 tablet_service.cc:1467] Processing CreateTablet for tablet 14cc3c4c39cb49e4a13114090de5a686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd5f6ace2e634c02887c259eebc09f7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:33.308145 10937 tablet_service.cc:1467] Processing CreateTablet for tablet 14cc3c4c39cb49e4a13114090de5a686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd5f6ace2e634c02887c259eebc09f7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:33.308377 11076 tablet_service.cc:1467] Processing CreateTablet for tablet 14cc3c4c39cb49e4a13114090de5a686 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cd5f6ace2e634c02887c259eebc09f7e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:06:33.309032 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cc3c4c39cb49e4a13114090de5a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.309043 11076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cc3c4c39cb49e4a13114090de5a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.309039 10798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cc3c4c39cb49e4a13114090de5a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.314296 11166 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: Bootstrap starting.
I20260925 11:06:33.315872 11166 tablet_bootstrap.cc:654] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.316124 11165 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: Bootstrap starting.
I20260925 11:06:33.317066 11167 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: Bootstrap starting.
I20260925 11:06:33.317919 11165 tablet_bootstrap.cc:654] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.318320 11166 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: No bootstrap required, opened a new log
I20260925 11:06:33.318445 11166 ts_tablet_manager.cc:1397] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 11:06:33.318753 11167 tablet_bootstrap.cc:654] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.319278 11166 raft_consensus.cc:348] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.319530 11166 raft_consensus.cc:374] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.319633 11166 raft_consensus.cc:729] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff9f32a7404dcb9adaf0214ece0d3c, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.319907 11165 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: No bootstrap required, opened a new log
I20260925 11:06:33.319862 11166 consensus_queue.cc:260] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.320086 11165 ts_tablet_manager.cc:1397] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260925 11:06:33.320510 11166 ts_tablet_manager.cc:1428] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 11:06:33.320348 11171 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.320765 11171 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.320820 11165 raft_consensus.cc:348] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.321065 11165 raft_consensus.cc:374] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.321163 11165 raft_consensus.cc:729] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.321429 11165 consensus_queue.cc:260] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.322029 11165 ts_tablet_manager.cc:1428] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 11:06:33.321957 11172 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.322364 11172 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.323668 11167 tablet_bootstrap.cc:492] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: No bootstrap required, opened a new log
I20260925 11:06:33.323815 11167 ts_tablet_manager.cc:1397] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260925 11:06:33.324671 11167 raft_consensus.cc:348] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.324906 11167 raft_consensus.cc:374] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.325011 11167 raft_consensus.cc:729] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.325282 11167 consensus_queue.cc:260] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.325826 11167 ts_tablet_manager.cc:1428] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:06:33.325677 11173 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.326007 11173 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.397173 11172 raft_consensus.cc:482] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:33.397409 11172 raft_consensus.cc:504] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.399294 11172 leader_election.cc:288] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:33.405927 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.399720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:43439 (local address 127.8.181.1:39165)
0925 11:06:33.399935 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.399954 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.400291 (+ 337us) server_negotiation.cc:407] Connection header received
0925 11:06:33.400529 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.400534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.400624 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.400728 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:33.401234 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.401993 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.404198 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.404983 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.405139 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.405246 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.405530 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.405656 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:06:33.406210 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.399717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42975 (local address 127.8.181.3:46663)
0925 11:06:33.399934 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.399939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.400766 (+ 827us) server_negotiation.cc:407] Connection header received
0925 11:06:33.400983 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.400990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.401052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.401152 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:33.401644 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.402306 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.404481 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.405255 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.405407 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.405521 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.405833 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.405956 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:06:33.406316 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.399494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46663 (local address 127.8.181.2:42975)
0925 11:06:33.400657 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.400671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.400837 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.401162 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.401166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.401188 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:33.401480 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.401488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.402438 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.402445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.404332 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.404343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.405493 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.405502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.405671 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.405693 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.405799 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.405922 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":510,"thread_start_us":364,"threads_started":1}
I20260925 11:06:33.406314 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.399480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:39165 (local address 127.8.181.2:43439)
0925 11:06:33.400168 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.400188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.400366 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.400742 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.400747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.400771 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:33.401083 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.401091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.402130 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.402135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.404046 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.404063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.405195 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.405202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.405358 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.405381 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.405508 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.405640 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":63,"thread_start_us":384,"threads_started":1}
I20260925 11:06:33.407070 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cc3c4c39cb49e4a13114090de5a686" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" is_pre_election: true
I20260925 11:06:33.407212 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cc3c4c39cb49e4a13114090de5a686" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5" is_pre_election: true
I20260925 11:06:33.407320 10818 raft_consensus.cc:2454] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:33.407580 11096 raft_consensus.cc:2454] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:33.407814 10889 leader_election.cc:302] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:33.408206 11172 raft_consensus.cc:2781] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:33.408329 11172 raft_consensus.cc:482] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:33.408428 11172 raft_consensus.cc:3037] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.410621 11172 raft_consensus.cc:504] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.411247 11172 leader_election.cc:288] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Requested vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:33.411689 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cc3c4c39cb49e4a13114090de5a686" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
I20260925 11:06:33.411739 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cc3c4c39cb49e4a13114090de5a686" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5"
I20260925 11:06:33.411913 10818 raft_consensus.cc:3037] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.411917 11096 raft_consensus.cc:3037] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.413905 10818 raft_consensus.cc:2454] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:33.413976 11096 raft_consensus.cc:2454] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:33.414345 10888 leader_election.cc:302] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:33.414669 11172 raft_consensus.cc:2781] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:33.415198 11172 raft_consensus.cc:686] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 LEADER]: Becoming Leader. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Running, Role: LEADER
I20260925 11:06:33.415694 11172 consensus_queue.cc:237] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.419363 11188 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: TxnStatusTablet state changed. Reason: New leader e49cf198c46c48048ba9d00b04c7cf6b. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.419823 11188 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:33.420590 11190 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 11:06:33.420873 11190 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 11:06:33.421665 10660 catalog_manager.cc:5495] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b reported cstate change: term changed from 0 to 1, leader changed from <none> to e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2). New cstate: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } health_report { overall_health: UNKNOWN } } }
I20260925 11:06:33.459672 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.452303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.0.0.1:56928)
0925 11:06:33.452592 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.452616 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.452777 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.453668 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.453672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.453683 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:33.453842 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.453850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.455509 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.455514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.456709 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.456723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.457756 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.457761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.457920 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.457983 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.459181 (+ 1198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.459207 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.459294 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.459421 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 11:06:33.459811 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.452306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.8.181.2:45249)
0925 11:06:33.453112 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.453119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.453140 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:06:33.453246 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.453253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.453409 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.453575 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:33.453990 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.455335 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.456868 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.457583 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.457724 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.458236 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.459041 (+ 805us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.459220 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.459351 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.459466 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655}
I20260925 11:06:33.468050 11096 raft_consensus.cc:1261] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.468075 10818 raft_consensus.cc:1261] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.469027 11188 consensus_queue.cc:1035] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.469360 11172 consensus_queue.cc:1035] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.482874 11173 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: TxnStatusTablet state changed. Reason: New leader e49cf198c46c48048ba9d00b04c7cf6b. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.483217 11173 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.484052 11171 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: TxnStatusTablet state changed. Reason: New leader e49cf198c46c48048ba9d00b04c7cf6b. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.484337 11171 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.488382 11172 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.488686 11172 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:33.496814 11172 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.497123 11172 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P e49cf198c46c48048ba9d00b04c7cf6b: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:06:33.498865 11196 mvcc.cc:205] Tried to move back new op lower bound from 7333209675635232768 to 7333209675434205184. Current Snapshot: MvccSnapshot[applied={T|T < 7333209675635232768}]
I20260925 11:06:33.500290 11173 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.500669 11173 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 85d00140c61b46aaabe57cb156aeb0f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.508673 11171 tablet_replica.cc:439] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } } }
I20260925 11:06:33.508991 11171 tablet_replica.cc:442] T 14cc3c4c39cb49e4a13114090de5a686 P 1aff9f32a7404dcb9adaf0214ece0d3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:06:33.516057 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.508120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.8.181.3:46663)
0925 11:06:33.508284 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.508290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.508406 (+ 116us) server_negotiation.cc:407] Connection header received
0925 11:06:33.508801 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.508806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.508918 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.509057 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:33.510970 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.512108 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.513511 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.514153 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.514280 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.514711 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.515423 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.515567 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.515622 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.515719 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 11:06:33.534267 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.527499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.8.181.3:58743)
0925 11:06:33.528390 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.528413 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.528545 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.528990 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.528994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.529005 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:33.529269 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.529277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.530860 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.530864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.532038 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.532052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.533046 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.533050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.533182 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.533245 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.533755 (+ 510us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.533773 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.533855 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.533965 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":616,"threads_started":1}
I20260925 11:06:33.534289 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.527572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:58743 (local address 127.8.181.2:45249)
0925 11:06:33.527772 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.527776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.528519 (+ 743us) server_negotiation.cc:407] Connection header received
0925 11:06:33.528684 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.528690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.528834 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.528937 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:33.529437 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.530723 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.532204 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.532860 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.533000 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.533464 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.533611 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.533739 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.533905 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.533987 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 11:06:33.544274 10818 raft_consensus.cc:1261] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Refusing update from remote peer 85d00140c61b46aaabe57cb156aeb0f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.544548 10962 raft_consensus.cc:1261] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Refusing update from remote peer 85d00140c61b46aaabe57cb156aeb0f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.544975 11175 consensus_queue.cc:1035] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.545382 11173 consensus_queue.cc:1035] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.555585 11193 mvcc.cc:205] Tried to move back new op lower bound from 7333209675950587904 to 7333209674542051328. Current Snapshot: MvccSnapshot[applied={T|T < 7333209675950587904}]
I20260925 11:06:33.585865 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.578681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:49385 (local address 127.8.181.3:46663)
0925 11:06:33.578857 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:33.578863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:33.579506 (+ 643us) server_negotiation.cc:407] Connection header received
0925 11:06:33.579781 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:33.579786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:33.579865 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:33.579926 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:33.580988 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.581690 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.583932 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.584592 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.584725 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.585176 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.585281 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.585376 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:33.585499 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 11:06:33.585573 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 11:06:33.586031 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:33.578710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46663 (local address 127.8.181.2:49385)
0925 11:06:33.579407 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:33.579432 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:33.579613 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:33.580480 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:33.580484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:33.580496 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:33.580845 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.580857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.581819 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.581825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.583784 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:33.583799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:33.584776 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:33.584785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:33.584933 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:33.584987 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:33.585377 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:33.585390 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 11:06:33.585480 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 11:06:33.585614 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":336,"threads_started":1}
I20260925 11:06:33.646608 10661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:33.646922 10661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:33.651265 10661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Sending DeleteTablet for 3 replicas of tablet c9858dfd957e46b8beca0c2c88e134e4
I20260925 11:06:33.652616 11076 tablet_service.cc:1514] Processing DeleteTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:33 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260925 11:06:33.653398 10937 tablet_service.cc:1514] Processing DeleteTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:33 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260925 11:06:33.653635 10798 tablet_service.cc:1514] Processing DeleteTablet for tablet c9858dfd957e46b8beca0c2c88e134e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:33 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:41058
I20260925 11:06:33.662735 10661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:33.665314 10661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:33.665868 11216 tablet_replica.cc:330] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: stopping tablet replica
I20260925 11:06:33.666392 11216 raft_consensus.cc:2229] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:33.666922 11216 raft_consensus.cc:2258] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:33.669235 11216 ts_tablet_manager.cc:1905] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:33.678084 10937 tablet_service.cc:1467] Processing CreateTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 (DEFAULT_TABLE table=test-table [id=4bc13502363946cdb19f4adc98b055dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:33.678676 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c592d7e73bfd416b84a570ea85aa7cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.683275 11075 tablet_service.cc:1467] Processing CreateTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 (DEFAULT_TABLE table=test-table [id=4bc13502363946cdb19f4adc98b055dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:33.683917 11075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c592d7e73bfd416b84a570ea85aa7cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.686693 11165 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Bootstrap starting.
I20260925 11:06:33.688575 11165 tablet_bootstrap.cc:654] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.691150 10797 tablet_service.cc:1467] Processing CreateTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 (DEFAULT_TABLE table=test-table [id=4bc13502363946cdb19f4adc98b055dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:33.691887 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c592d7e73bfd416b84a570ea85aa7cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:33.692606 11217 tablet_replica.cc:330] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: stopping tablet replica
I20260925 11:06:33.694574 11215 tablet_replica.cc:330] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: stopping tablet replica
I20260925 11:06:33.694909 11217 raft_consensus.cc:2229] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:33.695421 11215 raft_consensus.cc:2229] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:33.695847 11217 raft_consensus.cc:2258] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:33.696722 11215 raft_consensus.cc:2258] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:33.700170 11166 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Bootstrap starting.
I20260925 11:06:33.700462 11167 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Bootstrap starting.
I20260925 11:06:33.700433 11216 ts_tablet_manager.cc:1918] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:33.700623 11216 log.cc:1196] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c9858dfd957e46b8beca0c2c88e134e4
I20260925 11:06:33.701161 11216 ts_tablet_manager.cc:1939] T c9858dfd957e46b8beca0c2c88e134e4 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting consensus metadata
I20260925 11:06:33.701903 11166 tablet_bootstrap.cc:654] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.702101 11167 tablet_bootstrap.cc:654] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:33.706590 11165 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: No bootstrap required, opened a new log
I20260925 11:06:33.706771 11165 ts_tablet_manager.cc:1397] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20260925 11:06:33.707588 11165 raft_consensus.cc:348] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.707852 11165 raft_consensus.cc:374] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.707937 11165 raft_consensus.cc:729] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.708267 11165 consensus_queue.cc:260] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.709004 11165 ts_tablet_manager.cc:1428] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:06:33.710961 11167 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: No bootstrap required, opened a new log
I20260925 11:06:33.711169 11167 ts_tablet_manager.cc:1397] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260925 11:06:33.712154 11215 ts_tablet_manager.cc:1905] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:33.712193 11167 raft_consensus.cc:348] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.712438 11167 raft_consensus.cc:374] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.712517 11167 raft_consensus.cc:729] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.712836 11167 consensus_queue.cc:260] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.713654 11167 ts_tablet_manager.cc:1428] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:06:33.715306 11166 tablet_bootstrap.cc:492] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: No bootstrap required, opened a new log
I20260925 11:06:33.715521 11166 ts_tablet_manager.cc:1397] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260925 11:06:33.716229 11166 raft_consensus.cc:348] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.716517 11166 raft_consensus.cc:374] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:33.716617 11166 raft_consensus.cc:729] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff9f32a7404dcb9adaf0214ece0d3c, State: Initialized, Role: FOLLOWER
I20260925 11:06:33.716899 11166 consensus_queue.cc:260] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.718720 11166 ts_tablet_manager.cc:1428] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 11:06:33.719226 11217 ts_tablet_manager.cc:1905] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:33.719726 10647 catalog_manager.cc:4841] TS e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249): tablet c9858dfd957e46b8beca0c2c88e134e4 (table test-table [id=968e5c581d5049e5b96206b8282290f3]) successfully deleted
I20260925 11:06:33.726449 11217 ts_tablet_manager.cc:1918] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:33.726670 11217 log.cc:1196] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c9858dfd957e46b8beca0c2c88e134e4
I20260925 11:06:33.727250 11217 ts_tablet_manager.cc:1939] T c9858dfd957e46b8beca0c2c88e134e4 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting consensus metadata
I20260925 11:06:33.727605 11215 ts_tablet_manager.cc:1918] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:33.727859 11215 log.cc:1196] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c9858dfd957e46b8beca0c2c88e134e4
I20260925 11:06:33.729328 10646 catalog_manager.cc:4841] TS 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165): tablet c9858dfd957e46b8beca0c2c88e134e4 (table test-table [id=968e5c581d5049e5b96206b8282290f3]) successfully deleted
I20260925 11:06:33.729544 11215 ts_tablet_manager.cc:1939] T c9858dfd957e46b8beca0c2c88e134e4 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting consensus metadata
I20260925 11:06:33.731518 10645 catalog_manager.cc:4841] TS 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663): tablet c9858dfd957e46b8beca0c2c88e134e4 (table test-table [id=968e5c581d5049e5b96206b8282290f3]) successfully deleted
I20260925 11:06:33.781008 11210 raft_consensus.cc:482] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:33.781308 11210 raft_consensus.cc:504] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.782305 11210 leader_election.cc:288] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249)
I20260925 11:06:33.782886 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c592d7e73bfd416b84a570ea85aa7cf3" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" is_pre_election: true
I20260925 11:06:33.783171 10818 raft_consensus.cc:2454] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 0.
I20260925 11:06:33.783185 10961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c592d7e73bfd416b84a570ea85aa7cf3" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" is_pre_election: true
I20260925 11:06:33.783468 10961 raft_consensus.cc:2454] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 0.
I20260925 11:06:33.783697 11028 leader_election.cc:302] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c, 85d00140c61b46aaabe57cb156aeb0f5; no voters:
I20260925 11:06:33.784222 11210 raft_consensus.cc:2781] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:33.784344 11210 raft_consensus.cc:482] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:33.784444 11210 raft_consensus.cc:3037] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.787744 11210 raft_consensus.cc:504] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.788856 11210 leader_election.cc:288] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 election: Requested vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249)
I20260925 11:06:33.788997 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c592d7e73bfd416b84a570ea85aa7cf3" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
I20260925 11:06:33.789237 10818 raft_consensus.cc:3037] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.789276 10961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c592d7e73bfd416b84a570ea85aa7cf3" candidate_uuid: "85d00140c61b46aaabe57cb156aeb0f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49cf198c46c48048ba9d00b04c7cf6b"
I20260925 11:06:33.789480 10961 raft_consensus.cc:3037] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:33.792379 10818 raft_consensus.cc:2454] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 1.
I20260925 11:06:33.792912 10961 raft_consensus.cc:2454] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85d00140c61b46aaabe57cb156aeb0f5 in term 1.
I20260925 11:06:33.792959 11028 leader_election.cc:302] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c, 85d00140c61b46aaabe57cb156aeb0f5; no voters:
I20260925 11:06:33.793375 11210 raft_consensus.cc:2781] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:33.793911 11210 raft_consensus.cc:686] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 LEADER]: Becoming Leader. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Running, Role: LEADER
I20260925 11:06:33.794210 11210 consensus_queue.cc:237] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:33.822646 10661 catalog_manager.cc:5495] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3). New cstate: current_term: 1 leader_uuid: "85d00140c61b46aaabe57cb156aeb0f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } health_report { overall_health: HEALTHY } } }
I20260925 11:06:33.883942 10961 raft_consensus.cc:1261] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Refusing update from remote peer 85d00140c61b46aaabe57cb156aeb0f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.885083 10818 raft_consensus.cc:1261] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Refusing update from remote peer 85d00140c61b46aaabe57cb156aeb0f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:33.886068 11210 consensus_queue.cc:1035] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.887283 11209 consensus_queue.cc:1035] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:33.902076 11194 mvcc.cc:205] Tried to move back new op lower bound from 7333209677343502336 to 7333209676985966592. Current Snapshot: MvccSnapshot[applied={T|T < 7333209677343502336}]
W20260925 11:06:33.929533 11194 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 11:06:35.521234 10661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:35.521531 10661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:35.526080 10661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Sending DeleteTablet for 3 replicas of tablet c592d7e73bfd416b84a570ea85aa7cf3
I20260925 11:06:35.527029 10937 tablet_service.cc:1514] Processing DeleteTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260925 11:06:35.527029 10797 tablet_service.cc:1514] Processing DeleteTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:41058
I20260925 11:06:35.533059 11075 tablet_service.cc:1514] Processing DeleteTablet for tablet c592d7e73bfd416b84a570ea85aa7cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260925 11:06:35.533751 11249 tablet_replica.cc:330] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: stopping tablet replica
I20260925 11:06:35.534032 11249 raft_consensus.cc:2229] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:35.534528 11249 raft_consensus.cc:2258] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.535805 11249 ts_tablet_manager.cc:1905] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:35.537818 11247 tablet_replica.cc:330] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: stopping tablet replica
I20260925 11:06:35.538067 11247 raft_consensus.cc:2229] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:35.538399 11247 raft_consensus.cc:2258] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.539587 11247 ts_tablet_manager.cc:1905] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:35.541469 11246 tablet_replica.cc:330] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: stopping tablet replica
I20260925 11:06:35.541687 11246 raft_consensus.cc:2229] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:35.541921 11246 raft_consensus.cc:2258] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.542935 11246 ts_tablet_manager.cc:1905] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:35.543812 11249 ts_tablet_manager.cc:1918] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:35.544015 11249 log.cc:1196] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c592d7e73bfd416b84a570ea85aa7cf3
I20260925 11:06:35.544533 11249 ts_tablet_manager.cc:1939] T c592d7e73bfd416b84a570ea85aa7cf3 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting consensus metadata
I20260925 11:06:35.545761 10645 catalog_manager.cc:4841] TS 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663): tablet c592d7e73bfd416b84a570ea85aa7cf3 (table test-table [id=4bc13502363946cdb19f4adc98b055dd]) successfully deleted
I20260925 11:06:35.547519 11247 ts_tablet_manager.cc:1918] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:35.547680 11247 log.cc:1196] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c592d7e73bfd416b84a570ea85aa7cf3
I20260925 11:06:35.548076 11247 ts_tablet_manager.cc:1939] T c592d7e73bfd416b84a570ea85aa7cf3 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting consensus metadata
I20260925 11:06:35.549046 10646 catalog_manager.cc:4841] TS 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165): tablet c592d7e73bfd416b84a570ea85aa7cf3 (table test-table [id=4bc13502363946cdb19f4adc98b055dd]) successfully deleted
I20260925 11:06:35.550957 11246 ts_tablet_manager.cc:1918] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:35.551137 11246 log.cc:1196] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c592d7e73bfd416b84a570ea85aa7cf3
I20260925 11:06:35.551616 11246 ts_tablet_manager.cc:1939] T c592d7e73bfd416b84a570ea85aa7cf3 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting consensus metadata
I20260925 11:06:35.552827 10647 catalog_manager.cc:4841] TS e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249): tablet c592d7e73bfd416b84a570ea85aa7cf3 (table test-table [id=4bc13502363946cdb19f4adc98b055dd]) successfully deleted
Sep 25 11:06:35 dist-test-slave-6042 krb5kdc[10616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 11:06:35 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:35.589068 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.569321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.8.181.62:32837)
0925 11:06:35.569869 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:35.569873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:35.569898 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:06:35.570080 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:35.570084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:35.570150 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:35.570248 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:35.571604 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.572407 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.573789 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.574306 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.574419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.583734 (+ 9315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:35.583740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:35.583744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:35.583788 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:35.586548 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:35.587241 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:35.587246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:35.587248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:35.587391 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:35.587802 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:35.587806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:35.587808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:35.588116 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:35.588311 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:35.588611 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 11:06:35.588796 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260925 11:06:35.592854 10661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:35.593510 10661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:35.599836 10797 tablet_service.cc:1467] Processing CreateTablet for tablet a2434c2fbb76436cbece2dc1372122c8 (DEFAULT_TABLE table=test-table [id=f2980ef7164045cea1a2bbbdc0011e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:35.599987 10937 tablet_service.cc:1467] Processing CreateTablet for tablet a2434c2fbb76436cbece2dc1372122c8 (DEFAULT_TABLE table=test-table [id=f2980ef7164045cea1a2bbbdc0011e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:35.600126 11075 tablet_service.cc:1467] Processing CreateTablet for tablet a2434c2fbb76436cbece2dc1372122c8 (DEFAULT_TABLE table=test-table [id=f2980ef7164045cea1a2bbbdc0011e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:35.600375 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2434c2fbb76436cbece2dc1372122c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:35.600411 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2434c2fbb76436cbece2dc1372122c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:35.600509 11075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2434c2fbb76436cbece2dc1372122c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:35.607205 11258 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Bootstrap starting.
I20260925 11:06:35.607914 11259 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Bootstrap starting.
I20260925 11:06:35.610244 11259 tablet_bootstrap.cc:654] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:35.611936 11258 tablet_bootstrap.cc:654] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:35.612965 11259 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: No bootstrap required, opened a new log
I20260925 11:06:35.613251 11259 ts_tablet_manager.cc:1397] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:35.614331 11259 raft_consensus.cc:348] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.614682 11259 raft_consensus.cc:374] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:35.614786 11259 raft_consensus.cc:729] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Initialized, Role: FOLLOWER
I20260925 11:06:35.614960 11257 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Bootstrap starting.
I20260925 11:06:35.615031 11259 consensus_queue.cc:260] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.615371 11258 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: No bootstrap required, opened a new log
I20260925 11:06:35.615562 11258 ts_tablet_manager.cc:1397] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 11:06:35.615708 11259 ts_tablet_manager.cc:1428] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:06:35.616663 11257 tablet_bootstrap.cc:654] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:35.616680 11258 raft_consensus.cc:348] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.616928 11258 raft_consensus.cc:374] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:35.617023 11258 raft_consensus.cc:729] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Initialized, Role: FOLLOWER
I20260925 11:06:35.617332 11258 consensus_queue.cc:260] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.619381 11258 ts_tablet_manager.cc:1428] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:35.619750 11257 tablet_bootstrap.cc:492] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: No bootstrap required, opened a new log
I20260925 11:06:35.619912 11257 ts_tablet_manager.cc:1397] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 11:06:35.620633 11257 raft_consensus.cc:348] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.620867 11257 raft_consensus.cc:374] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:35.620975 11257 raft_consensus.cc:729] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff9f32a7404dcb9adaf0214ece0d3c, State: Initialized, Role: FOLLOWER
I20260925 11:06:35.621256 11257 consensus_queue.cc:260] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.622078 11257 ts_tablet_manager.cc:1428] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 11:06:35.840126 11241 raft_consensus.cc:482] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:35.840309 11241 raft_consensus.cc:504] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.840956 11241 leader_election.cc:288] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:35.841104 11263 raft_consensus.cc:482] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:35.841370 11263 raft_consensus.cc:504] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.841511 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" is_pre_election: true
I20260925 11:06:35.841569 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5" is_pre_election: true
I20260925 11:06:35.841804 11096 raft_consensus.cc:2454] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:35.842032 10818 raft_consensus.cc:2454] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:35.842320 10888 leader_election.cc:302] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:35.842633 11241 raft_consensus.cc:2781] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:35.842780 11241 raft_consensus.cc:482] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:35.842890 11241 raft_consensus.cc:3037] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:35.843667 11263 leader_election.cc:288] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:35.846740 11241 raft_consensus.cc:504] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.847515 11241 leader_election.cc:288] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Requested vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:35.848070 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
I20260925 11:06:35.848335 10818 raft_consensus.cc:3037] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:35.852626 10818 raft_consensus.cc:2454] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:35.853318 10889 leader_election.cc:302] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:35.853858 11241 raft_consensus.cc:2781] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:35.854595 11241 raft_consensus.cc:686] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 LEADER]: Becoming Leader. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Running, Role: LEADER
I20260925 11:06:35.855046 11241 consensus_queue.cc:237] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:35.860201 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5"
I20260925 11:06:35.860519 11096 raft_consensus.cc:3037] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:35.864984 11096 raft_consensus.cc:2454] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:35.864867 10661 catalog_manager.cc:5495] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b reported cstate change: term changed from 0 to 1, leader changed from <none> to e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2). New cstate: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } health_report { overall_health: UNKNOWN } } }
I20260925 11:06:35.868692 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.843930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.8.181.1:39743)
0925 11:06:35.861627 (+ 17697us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:35.861645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:35.861799 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:35.862206 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:35.862212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:35.862239 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:35.862685 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.862699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.864102 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.864107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.866210 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.866228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.867784 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.867790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.868014 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.868047 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:06:35.868158 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 11:06:35.868294 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17561,"mutex_wait_us":102,"spinlock_wait_cycles":768,"thread_start_us":16142,"threads_started":1}
I20260925 11:06:35.869416 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.844191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:39743 (local address 127.8.181.2:45249)
0925 11:06:35.859302 (+ 15111us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:35.859312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:35.861768 (+ 2456us) server_negotiation.cc:407] Connection header received
0925 11:06:35.861944 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:35.861949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:35.862049 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:35.862167 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:35.862877 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.863948 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.866391 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.867587 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.867734 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.867845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:35.868956 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0925 11:06:35.869119 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14940,"thread_start_us":14808,"threads_started":1}
I20260925 11:06:35.870031 10962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" is_pre_election: true
I20260925 11:06:35.871259 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.847800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:56371 (local address 127.8.181.3:46663)
0925 11:06:35.859861 (+ 12061us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:35.859866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:35.860779 (+ 913us) server_negotiation.cc:407] Connection header received
0925 11:06:35.860850 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:35.860855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:35.860924 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:35.861017 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:35.862185 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.863421 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.866344 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.867623 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.867744 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.867866 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:35.870797 (+ 2931us) server_negotiation.cc:299] Negotiation successful
0925 11:06:35.870936 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11931,"thread_start_us":11785,"threads_started":1}
I20260925 11:06:35.871317 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.843932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46663 (local address 127.8.181.1:56371)
0925 11:06:35.860464 (+ 16532us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:35.860488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:35.860651 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:35.861592 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:35.861613 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:35.861643 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:35.862000 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.862013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.863616 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.863623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.866164 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.866199 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.867995 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.868000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.870624 (+ 2624us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.870665 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:06:35.870771 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:06:35.870916 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16384,"mutex_wait_us":132,"spinlock_wait_cycles":25856,"thread_start_us":17361,"threads_started":1}
I20260925 11:06:35.871937 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2434c2fbb76436cbece2dc1372122c8" candidate_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5" is_pre_election: true
I20260925 11:06:35.872241 11096 raft_consensus.cc:2379] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1aff9f32a7404dcb9adaf0214ece0d3c in current term 1: Already voted for candidate e49cf198c46c48048ba9d00b04c7cf6b in this term.
I20260925 11:06:35.872741 10748 leader_election.cc:302] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c; no voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b
I20260925 11:06:35.873111 11263 raft_consensus.cc:2726] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 11:06:35.913125 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.908324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.8.181.2:45249)
0925 11:06:35.908513 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:35.908521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:35.908653 (+ 132us) server_negotiation.cc:407] Connection header received
0925 11:06:35.908853 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:35.908858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:35.908929 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:35.909024 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:35.909396 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.910097 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.911437 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.911872 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.912013 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.912437 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:35.912532 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:35.912629 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:35.912775 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 11:06:35.912849 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 11:06:35.926776 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.921366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:56905 (local address 127.8.181.2:45249)
0925 11:06:35.921548 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:35.921556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:35.922081 (+ 525us) server_negotiation.cc:407] Connection header received
0925 11:06:35.922177 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:35.922180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:35.922255 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:35.922330 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:35.922852 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.923593 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.924879 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.925324 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.925426 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.925849 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:35.925952 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:35.926031 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:35.926194 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 11:06:35.926335 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":54}
I20260925 11:06:35.926793 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:35.921338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45249 (local address 127.8.181.2:56905)
0925 11:06:35.921965 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:35.921981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:35.922093 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:35.922353 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:35.922356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:35.922364 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:06:35.922707 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.922717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.923732 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.923736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.924736 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:35.924747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:35.925488 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:35.925492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:35.925612 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:35.925677 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:06:35.926072 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:06:35.926082 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 11:06:35.926159 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 11:06:35.926261 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":405,"threads_started":1}
I20260925 11:06:35.935467 11096 raft_consensus.cc:1261] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:35.935521 10818 raft_consensus.cc:1261] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:35.936045 11198 consensus_queue.cc:1035] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:35.936381 11241 consensus_queue.cc:1035] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:35.982381 10660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:35.982743 10660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:35.985914 10660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Sending DeleteTablet for 3 replicas of tablet a2434c2fbb76436cbece2dc1372122c8
I20260925 11:06:35.987040 10797 tablet_service.cc:1514] Processing DeleteTablet for tablet a2434c2fbb76436cbece2dc1372122c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:41058
I20260925 11:06:35.987314 11247 tablet_replica.cc:330] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: stopping tablet replica
I20260925 11:06:35.987357 11075 tablet_service.cc:1514] Processing DeleteTablet for tablet a2434c2fbb76436cbece2dc1372122c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260925 11:06:35.987524 11247 raft_consensus.cc:2229] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:35.987720 11247 raft_consensus.cc:2258] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.987985 11249 tablet_replica.cc:330] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: stopping tablet replica
I20260925 11:06:35.988195 11249 raft_consensus.cc:2229] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:35.988397 11249 raft_consensus.cc:2258] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.988516 10937 tablet_service.cc:1514] Processing DeleteTablet for tablet a2434c2fbb76436cbece2dc1372122c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:35 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260925 11:06:35.988817 11246 tablet_replica.cc:330] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: stopping tablet replica
I20260925 11:06:35.988894 11247 ts_tablet_manager.cc:1905] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:35.989029 11246 raft_consensus.cc:2229] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:35.989497 11246 raft_consensus.cc:2258] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:35.990203 10660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 11:06:35.990798 11246 ts_tablet_manager.cc:1905] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 11:06:35.991171 10660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:35.992758 11249 ts_tablet_manager.cc:1905] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:35.996932 11246 ts_tablet_manager.cc:1918] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:35.997203 11246 log.cc:1196] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/a2434c2fbb76436cbece2dc1372122c8
I20260925 11:06:35.997749 11246 ts_tablet_manager.cc:1939] T a2434c2fbb76436cbece2dc1372122c8 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting consensus metadata
I20260925 11:06:35.999043 10647 catalog_manager.cc:4841] TS e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249): tablet a2434c2fbb76436cbece2dc1372122c8 (table test-table [id=f2980ef7164045cea1a2bbbdc0011e11]) successfully deleted
I20260925 11:06:36.000563 11247 ts_tablet_manager.cc:1918] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:36.000744 11247 log.cc:1196] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/a2434c2fbb76436cbece2dc1372122c8
I20260925 11:06:36.001084 11247 ts_tablet_manager.cc:1939] T a2434c2fbb76436cbece2dc1372122c8 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting consensus metadata
I20260925 11:06:36.001662 11249 ts_tablet_manager.cc:1918] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:06:36.001816 11249 log.cc:1196] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/a2434c2fbb76436cbece2dc1372122c8
I20260925 11:06:36.002260 11249 ts_tablet_manager.cc:1939] T a2434c2fbb76436cbece2dc1372122c8 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting consensus metadata
I20260925 11:06:36.002252 10646 catalog_manager.cc:4841] TS 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165): tablet a2434c2fbb76436cbece2dc1372122c8 (table test-table [id=f2980ef7164045cea1a2bbbdc0011e11]) successfully deleted
I20260925 11:06:36.003376 10645 catalog_manager.cc:4841] TS 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663): tablet a2434c2fbb76436cbece2dc1372122c8 (table test-table [id=f2980ef7164045cea1a2bbbdc0011e11]) successfully deleted
I20260925 11:06:36.006644 10797 tablet_service.cc:1467] Processing CreateTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 (DEFAULT_TABLE table=test-table [id=26876b3471b64d9e8f27ba98f96ee667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:36.006819 10937 tablet_service.cc:1467] Processing CreateTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 (DEFAULT_TABLE table=test-table [id=26876b3471b64d9e8f27ba98f96ee667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:36.007100 10797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e8f769fd644ce1a0f4264b88303cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:36.007313 10937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e8f769fd644ce1a0f4264b88303cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:36.011809 11075 tablet_service.cc:1467] Processing CreateTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 (DEFAULT_TABLE table=test-table [id=26876b3471b64d9e8f27ba98f96ee667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:36.012213 11075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e8f769fd644ce1a0f4264b88303cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:36.012800 11259 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Bootstrap starting.
I20260925 11:06:36.013551 11257 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Bootstrap starting.
I20260925 11:06:36.014655 11259 tablet_bootstrap.cc:654] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:36.015265 11257 tablet_bootstrap.cc:654] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:36.021293 11258 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Bootstrap starting.
I20260925 11:06:36.023669 11258 tablet_bootstrap.cc:654] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:36.024238 11259 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: No bootstrap required, opened a new log
I20260925 11:06:36.024430 11259 ts_tablet_manager.cc:1397] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260925 11:06:36.025170 11257 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: No bootstrap required, opened a new log
I20260925 11:06:36.025462 11257 ts_tablet_manager.cc:1397] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260925 11:06:36.025709 11259 raft_consensus.cc:348] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.025995 11259 raft_consensus.cc:374] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:36.026072 11259 raft_consensus.cc:729] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Initialized, Role: FOLLOWER
I20260925 11:06:36.026371 11259 consensus_queue.cc:260] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.026770 11257 raft_consensus.cc:348] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.027150 11257 raft_consensus.cc:374] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:36.027271 11257 raft_consensus.cc:729] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff9f32a7404dcb9adaf0214ece0d3c, State: Initialized, Role: FOLLOWER
I20260925 11:06:36.027698 11257 consensus_queue.cc:260] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.028543 11257 ts_tablet_manager.cc:1428] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 11:06:36.029526 11259 ts_tablet_manager.cc:1428] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 11:06:36.037904 11258 tablet_bootstrap.cc:492] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: No bootstrap required, opened a new log
I20260925 11:06:36.038113 11258 ts_tablet_manager.cc:1397] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260925 11:06:36.038874 11258 raft_consensus.cc:348] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.039108 11258 raft_consensus.cc:374] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:36.039191 11258 raft_consensus.cc:729] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85d00140c61b46aaabe57cb156aeb0f5, State: Initialized, Role: FOLLOWER
I20260925 11:06:36.039458 11258 consensus_queue.cc:260] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.040836 11258 ts_tablet_manager.cc:1428] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:36.063407 11231 raft_consensus.cc:482] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:36.063647 11231 raft_consensus.cc:504] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.064471 11231 leader_election.cc:288] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:36.065002 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e8f769fd644ce1a0f4264b88303cf1" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5" is_pre_election: true
I20260925 11:06:36.065002 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e8f769fd644ce1a0f4264b88303cf1" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" is_pre_election: true
I20260925 11:06:36.065263 11096 raft_consensus.cc:2454] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:36.065301 10818 raft_consensus.cc:2454] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 0.
I20260925 11:06:36.065759 10888 leader_election.cc:302] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1aff9f32a7404dcb9adaf0214ece0d3c, 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:36.066139 11231 raft_consensus.cc:2781] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:36.066274 11231 raft_consensus.cc:482] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:36.066429 11231 raft_consensus.cc:3037] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:36.069626 11231 raft_consensus.cc:504] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.070389 11231 leader_election.cc:288] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Requested vote from peers 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165), 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663)
I20260925 11:06:36.070941 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e8f769fd644ce1a0f4264b88303cf1" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85d00140c61b46aaabe57cb156aeb0f5"
I20260925 11:06:36.070976 10818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e8f769fd644ce1a0f4264b88303cf1" candidate_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c"
I20260925 11:06:36.071197 11096 raft_consensus.cc:3037] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:36.071244 10818 raft_consensus.cc:3037] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:36.073410 11096 raft_consensus.cc:2454] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:36.073946 10888 leader_election.cc:302] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85d00140c61b46aaabe57cb156aeb0f5, e49cf198c46c48048ba9d00b04c7cf6b; no voters:
I20260925 11:06:36.074250 10818 raft_consensus.cc:2454] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49cf198c46c48048ba9d00b04c7cf6b in term 1.
I20260925 11:06:36.074643 11231 raft_consensus.cc:2781] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:36.074970 11231 raft_consensus.cc:686] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 LEADER]: Becoming Leader. State: Replica: e49cf198c46c48048ba9d00b04c7cf6b, State: Running, Role: LEADER
I20260925 11:06:36.075275 11231 consensus_queue.cc:237] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } }
I20260925 11:06:36.078171 10661 catalog_manager.cc:5495] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b reported cstate change: term changed from 0 to 1, leader changed from <none> to e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2). New cstate: current_term: 1 leader_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49cf198c46c48048ba9d00b04c7cf6b" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 } health_report { overall_health: UNKNOWN } } }
I20260925 11:06:36.110064 10818 raft_consensus.cc:1261] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:36.110277 11096 raft_consensus.cc:1261] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Refusing update from remote peer e49cf198c46c48048ba9d00b04c7cf6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:36.110670 11198 consensus_queue.cc:1035] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff9f32a7404dcb9adaf0214ece0d3c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:36.110944 11241 consensus_queue.cc:1035] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85d00140c61b46aaabe57cb156aeb0f5" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 11:06:36.135847 11224 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 11:06:37.721464 10661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:37.721740 10661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:37.725672 10661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89c1f4ef1c434f62a5830cfb0fac3470: Sending DeleteTablet for 3 replicas of tablet c7e8f769fd644ce1a0f4264b88303cf1
I20260925 11:06:37.726568 10937 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:56912
I20260925 11:06:37.726594 10797 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:41058
I20260925 11:06:37.726727 11075 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e8f769fd644ce1a0f4264b88303cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53238
I20260925 11:06:37.727617 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10732
I20260925 11:06:37.736863 11301 tablet_replica.cc:330] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: stopping tablet replica
I20260925 11:06:37.737035 11303 tablet_replica.cc:330] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: stopping tablet replica
I20260925 11:06:37.737262 11301 raft_consensus.cc:2229] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:37.737409 11303 raft_consensus.cc:2229] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:37.737739 11303 raft_consensus.cc:2258] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:37.737815 11301 raft_consensus.cc:2258] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:37.739108 11301 ts_tablet_manager.cc:1905] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:37.739120 11303 ts_tablet_manager.cc:1905] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:37.745981 11303 ts_tablet_manager.cc:1918] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:37.746196 11303 log.cc:1196] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c7e8f769fd644ce1a0f4264b88303cf1
I20260925 11:06:37.746362 11302 tablet_replica.cc:330] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: stopping tablet replica
I20260925 11:06:37.746706 11303 ts_tablet_manager.cc:1939] T c7e8f769fd644ce1a0f4264b88303cf1 P 85d00140c61b46aaabe57cb156aeb0f5: Deleting consensus metadata
I20260925 11:06:37.746708 11302 raft_consensus.cc:2229] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:37.747112 11302 raft_consensus.cc:2258] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 11:06:37 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:37.747861 10645 catalog_manager.cc:4841] TS 85d00140c61b46aaabe57cb156aeb0f5 (127.8.181.3:46663): tablet c7e8f769fd644ce1a0f4264b88303cf1 (table test-table [id=26876b3471b64d9e8f27ba98f96ee667]) successfully deleted
I20260925 11:06:37.748270 11302 ts_tablet_manager.cc:1905] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:37.748781 11301 ts_tablet_manager.cc:1918] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:37.748934 11301 log.cc:1196] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c7e8f769fd644ce1a0f4264b88303cf1
I20260925 11:06:37.749351 11301 ts_tablet_manager.cc:1939] T c7e8f769fd644ce1a0f4264b88303cf1 P e49cf198c46c48048ba9d00b04c7cf6b: Deleting consensus metadata
I20260925 11:06:37.750600 10647 catalog_manager.cc:4841] TS e49cf198c46c48048ba9d00b04c7cf6b (127.8.181.2:45249): tablet c7e8f769fd644ce1a0f4264b88303cf1 (table test-table [id=26876b3471b64d9e8f27ba98f96ee667]) successfully deleted
I20260925 11:06:37.754659 11302 ts_tablet_manager.cc:1918] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 11:06:37.754825 11302 log.cc:1196] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c7e8f769fd644ce1a0f4264b88303cf1
I20260925 11:06:37.755298 11302 ts_tablet_manager.cc:1939] T c7e8f769fd644ce1a0f4264b88303cf1 P 1aff9f32a7404dcb9adaf0214ece0d3c: Deleting consensus metadata
I20260925 11:06:37.755702 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:37.730075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.8.181.1:39165)
0925 11:06:37.735807 (+ 5732us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:37.735817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:37.735849 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:06:37.735933 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:37.735941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:37.736046 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:37.736198 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:37.738981 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.740123 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.742067 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.742751 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.742866 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:37.748788 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:37.748809 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:37.748832 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:37.748880 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:37.752383 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.753250 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.753256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.753259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.753389 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.753818 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.753821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.753823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.754229 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:37.754451 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:37.754994 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 11:06:37.755262 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5543,"thread_start_us":5365,"threads_started":1}
I20260925 11:06:37.756469 10646 catalog_manager.cc:4841] TS 1aff9f32a7404dcb9adaf0214ece0d3c (127.8.181.1:39165): tablet c7e8f769fd644ce1a0f4264b88303cf1 (table test-table [id=26876b3471b64d9e8f27ba98f96ee667]) successfully deleted
W20260925 11:06:37.756852 10863 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41062
W20260925 11:06:37.757459 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:37.757678 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10732
I20260925 11:06:37.776292 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10872
Sep 25 11:06:37 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:37.791545 11306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:37.777227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.8.181.2:45249)
0925 11:06:37.777730 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:37.777734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:37.777751 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:06:37.777803 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:37.777807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:37.777864 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:37.777956 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:37.779248 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.780035 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.781238 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.781649 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.781760 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:37.786002 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:37.786017 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:37.786033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:37.786068 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:37.789043 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.789691 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.789698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.789701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.789818 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.790149 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.790152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.790154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.790604 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:37.790825 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:37.791096 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 11:06:37.791283 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":270,"threads_started":1}
W20260925 11:06:37.792264 11002 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56948
W20260925 11:06:37.792994 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:37.793134 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10872
I20260925 11:06:37.818885 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11011
Sep 25 11:06:37 dist-test-slave-6042 krb5kdc[10616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:37.836369 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:37.819773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.8.181.3:46663)
0925 11:06:37.820366 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:37.820372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:37.820392 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:06:37.820459 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:37.820465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:37.820556 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:37.820690 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:37.822134 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.823037 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.824405 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:37.824849 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:37.825014 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:37.830075 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:37.830091 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:37.830106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:37.830144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:37.833180 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.833952 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.833959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.833961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.834081 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:37.834492 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:37.834499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:37.834503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:37.834865 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:37.835387 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:37.835769 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 11:06:37.836012 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
W20260925 11:06:37.837262 11141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53268
W20260925 11:06:37.837970 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11011: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:37.838142 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11011
I20260925 11:06:37.861136 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 10629
I20260925 11:06:37.862353 10701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:37.983938 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 10629
2026-09-25T11:06:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8618 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:38 dist-test-slave-6042 krb5kdc[11311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:38 dist-test-slave-6042 krb5kdc[11311](info): set up 2 sockets
Sep 25 11:06:38 dist-test-slave-6042 krb5kdc[11311](info): commencing operation
krb5kdc: starting...
W20260925 11:06:39.111096 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:39 dist-test-slave-6042 krb5kdc[11311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:39.319660 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:42267
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:39.659699 11324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:39.660020 11324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:39.660192 11324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:39.667732 11324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:39.667857 11324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:39.667955 11324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:39.668036 11324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:39.668097 11324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:39.678664 11324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:42267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:39.680145 11324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:39.681530 11324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:39.694841 11329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:39.694749 11332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:39.694782 11330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:39.695339 11324 server_base.cc:1034] running on GCE node
I20260925 11:06:39.696512 11324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:39.698230 11324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:39.699599 11324 hybrid_clock.cc:648] HybridClock initialized: now 1790334399699551 us; error 75 us; skew 500 ppm
I20260925 11:06:39.699937 11324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:39 dist-test-slave-6042 krb5kdc[11311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334399, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:39.703656 11324 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:39.705449 11324 webserver.cc:460] Webserver started at http://127.8.181.62:42387/ using document root <none> and password file <none>
I20260925 11:06:39.706204 11324 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:39.706311 11324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:39.706708 11324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:39.710013 11324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "778c0ae9068442b287b075be64cf0b49"
format_stamp: "Formatted at 2026-09-25 11:06:39 on dist-test-slave-6042"
server_key: "aa422e0881d2016433127f32860abe98"
server_key_iv: "d7e60e66e1ba551de3a419364308d2da"
server_key_version: "encryptionkey@0"
I20260925 11:06:39.710947 11324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "778c0ae9068442b287b075be64cf0b49"
format_stamp: "Formatted at 2026-09-25 11:06:39 on dist-test-slave-6042"
server_key: "aa422e0881d2016433127f32860abe98"
server_key_iv: "d7e60e66e1ba551de3a419364308d2da"
server_key_version: "encryptionkey@0"
I20260925 11:06:39.716512 11324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:06:39.720367 11339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:39.721925 11324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:06:39.722136 11324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "778c0ae9068442b287b075be64cf0b49"
format_stamp: "Formatted at 2026-09-25 11:06:39 on dist-test-slave-6042"
server_key: "aa422e0881d2016433127f32860abe98"
server_key_iv: "d7e60e66e1ba551de3a419364308d2da"
server_key_version: "encryptionkey@0"
I20260925 11:06:39.722357 11324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:39.747603 11324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:39.752580 11324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:39.752902 11324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:39.772933 11324 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:42267
I20260925 11:06:39.772977 11390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:42267 every 8 connection(s)
I20260925 11:06:39.774391 11324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:39.778210 11391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:39.780179 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11324
I20260925 11:06:39.780452 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:39.781144 8916 external_mini_cluster.cc:1428] Setting key 80680422abf82b4e19385518ac2094b2
I20260925 11:06:39.791883 11391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49: Bootstrap starting.
I20260925 11:06:39.795658 11391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:39.796713 11391 log.cc:824] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:06:39 dist-test-slave-6042 krb5kdc[11311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:39.800532 11391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49: No bootstrap required, opened a new log
I20260925 11:06:39.807977 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:39.783519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.8.181.62:42267)
0925 11:06:39.784389 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:39.784406 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:39.784461 (+ 55us) server_negotiation.cc:407] Connection header received
0925 11:06:39.786841 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:39.786871 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:39.787416 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:39.787938 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:39.789323 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:39.791358 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:39.792816 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:39.793946 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:39.794168 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:39.798549 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:39.798561 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:39.798582 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:39.798631 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:39.802832 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:39.803652 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:39.803661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:39.803671 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:39.803900 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:39.804331 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:39.804336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:39.804339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:39.805148 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:39.805479 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:39.805864 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 11:06:39.806408 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":337,"threads_started":1}
I20260925 11:06:39.811443 11391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } }
I20260925 11:06:39.811832 11391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:39.811906 11391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778c0ae9068442b287b075be64cf0b49, State: Initialized, Role: FOLLOWER
I20260925 11:06:39.812394 11391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } }
I20260925 11:06:39.812590 11391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:39.812692 11391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:39.812812 11391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:39.815047 11391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } }
I20260925 11:06:39.815645 11391 leader_election.cc:302] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778c0ae9068442b287b075be64cf0b49; no voters:
I20260925 11:06:39.816274 11391 leader_election.cc:288] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:39.816430 11396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:39.817157 11396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [term 1 LEADER]: Becoming Leader. State: Replica: 778c0ae9068442b287b075be64cf0b49, State: Running, Role: LEADER
I20260925 11:06:39.817682 11396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } }
I20260925 11:06:39.818394 11391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:39.820565 11397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778c0ae9068442b287b075be64cf0b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } } }
I20260925 11:06:39.821039 11397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:39.820556 11398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778c0ae9068442b287b075be64cf0b49. Latest consensus state: current_term: 1 leader_uuid: "778c0ae9068442b287b075be64cf0b49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778c0ae9068442b287b075be64cf0b49" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 42267 } } }
I20260925 11:06:39.821530 11398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:39.823925 11401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:39.829774 11401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:39.840219 11401 catalog_manager.cc:1347] Generated new cluster ID: 82afa0d592f64f5e87bfac1fa24cd0e4
I20260925 11:06:39.840365 11401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:39.855000 11401 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:39.856454 11401 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:39.869867 11401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778c0ae9068442b287b075be64cf0b49: Generated new TSK 0
I20260925 11:06:39.871227 11401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:39.935220 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:40.277983 11417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:40.278281 11417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:40.278447 11417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:40.286422 11417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:40.286650 11417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:40.286852 11417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:40.297374 11417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:40.298885 11417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:40.300237 11417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:40.314581 11422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:40.314817 11423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:40.314616 11425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:40.315744 11417 server_base.cc:1034] running on GCE node
I20260925 11:06:40.316388 11417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:40.317438 11417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:40.318691 11417 hybrid_clock.cc:648] HybridClock initialized: now 1790334400318671 us; error 90 us; skew 500 ppm
I20260925 11:06:40.319046 11417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:40 dist-test-slave-6042 krb5kdc[11311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334400, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:40.322747 11417 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:40.324383 11417 webserver.cc:460] Webserver started at http://127.8.181.1:37693/ using document root <none> and password file <none>
I20260925 11:06:40.325101 11417 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:40.325239 11417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:40.325591 11417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:40.328754 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "7bfd17b4cef745bda76855dc6c749c2b"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "b91bd3493e051eb0f227a86ca167b71f"
server_key_iv: "a1b59d007637c561a4d1c529307a2161"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.329569 11417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "7bfd17b4cef745bda76855dc6c749c2b"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "b91bd3493e051eb0f227a86ca167b71f"
server_key_iv: "a1b59d007637c561a4d1c529307a2161"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.335194 11417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 11:06:40.339043 11432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.340497 11417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:06:40.340687 11417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "7bfd17b4cef745bda76855dc6c749c2b"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "b91bd3493e051eb0f227a86ca167b71f"
server_key_iv: "a1b59d007637c561a4d1c529307a2161"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.340899 11417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:40.360503 11417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:40.365437 11417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:40.365764 11417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:40.366882 11417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:40.368346 11417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:40.368460 11417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.368637 11417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:40.368709 11417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.403345 11417 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:41053
I20260925 11:06:40.403391 11544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:41053 every 8 connection(s)
I20260925 11:06:40.404807 11417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:40.407958 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11417
I20260925 11:06:40.408197 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:40.408795 8916 external_mini_cluster.cc:1428] Setting key 9331f963142f349ad80d82468b4d9d35
Sep 25 11:06:40 dist-test-slave-6042 krb5kdc[11311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334400, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:06:40.430176 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:40.407585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:49443 (local address 127.8.181.62:42267)
0925 11:06:40.408136 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:40.408145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:40.409338 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 11:06:40.411694 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:40.411698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:40.411773 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:40.411860 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:40.414144 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.415012 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.417482 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.418252 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.418404 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:40.422551 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:40.422561 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:40.422569 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:40.422628 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:40.426511 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:40.427523 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:40.427532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:40.427538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:40.427726 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:40.428170 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:40.428177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:40.428181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:40.428587 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:40.428784 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:40.429496 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 11:06:40.429784 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260925 11:06:40.431012 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:40.408218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:42267 (local address 127.8.181.1:49443)
0925 11:06:40.409148 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:40.409198 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:40.411354 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:40.412176 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:40.412191 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:40.413061 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:40.413932 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:40.413961 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.415201 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.415210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:40.417324 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:40.417338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.418519 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.418529 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:40.418809 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:40.419598 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:40.419638 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:40.422235 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:40.426735 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:40.426747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:40.426770 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:40.427353 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:40.427848 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:40.427851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:40.427853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:40.428050 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:40.428732 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:40.428739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:40.429092 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 11:06:40.429341 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 11:06:40.429764 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":393,"threads_started":1}
I20260925 11:06:40.432744 11545 heartbeater.cc:347] Connected to a master server at 127.8.181.62:42267
I20260925 11:06:40.433137 11545 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:40.433957 11545 heartbeater.cc:511] Master 127.8.181.62:42267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:06:40.436236 11356 ts_manager.cc:194] Registered new tserver with Master: 7bfd17b4cef745bda76855dc6c749c2b (127.8.181.1:41053)
I20260925 11:06:40.437731 11356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:49443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:40.462051 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:40.805791 11551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:40.806087 11551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:40.806181 11551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:40.813784 11551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:40.813912 11551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:40.814029 11551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:40.824622 11551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:40.825912 11551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:40.827297 11551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:40.842974 11556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:40.843031 11559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:40.843175 11557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:40.844003 11551 server_base.cc:1034] running on GCE node
I20260925 11:06:40.844717 11551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:40.845901 11551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:40.847160 11551 hybrid_clock.cc:648] HybridClock initialized: now 1790334400847079 us; error 54 us; skew 500 ppm
I20260925 11:06:40.847502 11551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:40 dist-test-slave-6042 krb5kdc[11311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334400, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:40.856597 11551 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:40.858306 11551 webserver.cc:460] Webserver started at http://127.8.181.2:36811/ using document root <none> and password file <none>
I20260925 11:06:40.859138 11551 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:40.859299 11551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:40.859630 11551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:40.862757 11551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "673871bb410a4f2396d584f204437631"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "96865212c7dc3f3e95c517b8fb0e8fc2"
server_key_iv: "6012ed37a5a92ae67c8419ead78f14c7"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.863569 11551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "673871bb410a4f2396d584f204437631"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "96865212c7dc3f3e95c517b8fb0e8fc2"
server_key_iv: "6012ed37a5a92ae67c8419ead78f14c7"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.868930 11551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 11:06:40.872918 11566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.874362 11551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:40.874598 11551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "673871bb410a4f2396d584f204437631"
format_stamp: "Formatted at 2026-09-25 11:06:40 on dist-test-slave-6042"
server_key: "96865212c7dc3f3e95c517b8fb0e8fc2"
server_key_iv: "6012ed37a5a92ae67c8419ead78f14c7"
server_key_version: "encryptionkey@0"
I20260925 11:06:40.874871 11551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:40.896561 11551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:40.901331 11551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:40.901679 11551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:40.902716 11551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:40.904258 11551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:40.904369 11551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.904525 11551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:40.904596 11551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:40.939733 11551 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:39591
I20260925 11:06:40.939831 11678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:39591 every 8 connection(s)
I20260925 11:06:40.941118 11551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:40.942606 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11551
I20260925 11:06:40.942835 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:40.943321 8916 external_mini_cluster.cc:1428] Setting key bcac7838edf61514bfef3d92d124a5e8
Sep 25 11:06:40 dist-test-slave-6042 krb5kdc[11311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334400, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:40.964743 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:40.944025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:59825 (local address 127.8.181.62:42267)
0925 11:06:40.944677 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:40.944682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:40.945798 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 11:06:40.947752 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:40.947758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:40.947847 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:40.948005 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:40.949619 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.950398 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.952059 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.952852 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.953003 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:40.957165 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:40.957176 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:40.957184 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:40.957239 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:40.961194 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:40.962061 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:40.962071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:40.962076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:40.962304 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:40.962889 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:40.962896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:40.962902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:40.963339 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:40.963516 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:40.964289 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 11:06:40.964467 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":409,"threads_started":1}
I20260925 11:06:40.965866 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:40.944752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:42267 (local address 127.8.181.2:59825)
0925 11:06:40.945627 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:40.945696 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:40.947459 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:40.948260 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:40.948270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:40.948822 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:40.949421 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:40.949440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.950619 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.950626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:40.951904 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:40.951916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:40.953157 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:40.953173 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:40.953393 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:40.954169 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:40.954198 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:40.956827 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:40.961404 (+ 4577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:40.961414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:40.961431 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:40.961916 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:40.962423 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:40.962427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:40.962429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:40.962764 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:40.963476 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:40.963482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:40.963921 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 11:06:40.964170 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 11:06:40.964586 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":345,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:06:40.967589 11679 heartbeater.cc:347] Connected to a master server at 127.8.181.62:42267
I20260925 11:06:40.967923 11679 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:40.968881 11679 heartbeater.cc:511] Master 127.8.181.62:42267 requested a full tablet report, sending...
I20260925 11:06:40.970772 11356 ts_manager.cc:194] Registered new tserver with Master: 673871bb410a4f2396d584f204437631 (127.8.181.2:39591)
I20260925 11:06:40.971659 11356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:59825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:40.995199 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:41.331905 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:41.332203 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:41.332361 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:41.340153 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:41.340279 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:41.340468 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:41.350941 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:42267
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:41.352291 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:41.353603 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:41.367763 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:41.367663 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:41.367805 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:41.368809 11685 server_base.cc:1034] running on GCE node
I20260925 11:06:41.369488 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:41.370676 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:41.371888 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790334401371850 us; error 28 us; skew 500 ppm
I20260925 11:06:41.372328 11685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:41 dist-test-slave-6042 krb5kdc[11311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:41.375669 11685 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:41.377300 11685 webserver.cc:460] Webserver started at http://127.8.181.3:43675/ using document root <none> and password file <none>
I20260925 11:06:41.378068 11685 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:41.378228 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:41.378639 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:41.381736 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "2ec0b186316b443ca2d125f87879b8d2"
format_stamp: "Formatted at 2026-09-25 11:06:41 on dist-test-slave-6042"
server_key: "78122c3e85b15cde8d1308ab28bf9004"
server_key_iv: "f3f4959a7e69b00a63fa1a69aa0cc31b"
server_key_version: "encryptionkey@0"
I20260925 11:06:41.382637 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "2ec0b186316b443ca2d125f87879b8d2"
format_stamp: "Formatted at 2026-09-25 11:06:41 on dist-test-slave-6042"
server_key: "78122c3e85b15cde8d1308ab28bf9004"
server_key_iv: "f3f4959a7e69b00a63fa1a69aa0cc31b"
server_key_version: "encryptionkey@0"
I20260925 11:06:41.388033 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 11:06:41.391705 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:41.393168 11685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:06:41.393366 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "2ec0b186316b443ca2d125f87879b8d2"
format_stamp: "Formatted at 2026-09-25 11:06:41 on dist-test-slave-6042"
server_key: "78122c3e85b15cde8d1308ab28bf9004"
server_key_iv: "f3f4959a7e69b00a63fa1a69aa0cc31b"
server_key_version: "encryptionkey@0"
I20260925 11:06:41.393584 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:41.416887 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:41.421954 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:41.422283 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:41.423416 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:41.425007 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:41.425119 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:41.425282 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:41.425359 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:41.440919 11545 heartbeater.cc:503] Master 127.8.181.62:42267 was elected leader, sending a full tablet report...
I20260925 11:06:41.459303 11685 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:41737
I20260925 11:06:41.459335 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:41737 every 8 connection(s)
I20260925 11:06:41.460708 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:41.466238 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11685
I20260925 11:06:41.466492 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:41.466974 8916 external_mini_cluster.cc:1428] Setting key 52380614af9b76f4a73922810295ba2e
Sep 25 11:06:41 dist-test-slave-6042 krb5kdc[11311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:41.482363 11681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:41.463586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:48005 (local address 127.8.181.62:42267)
0925 11:06:41.463831 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:41.463836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:41.464806 (+ 970us) server_negotiation.cc:407] Connection header received
0925 11:06:41.466716 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:41.466723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:41.466828 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:41.466944 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:41.468864 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:41.469763 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:41.471502 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:41.471913 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:41.472053 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:41.475556 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:41.475562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:41.475565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:41.475600 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:41.478684 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:41.479668 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:41.479672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:41.479675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:41.479806 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:41.480278 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:41.480283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:41.480285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:41.480596 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:41.480771 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:41.481714 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 11:06:41.482038 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 11:06:41.483462 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:41.463914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:42267 (local address 127.8.181.3:48005)
0925 11:06:41.464640 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:41.464681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:41.466337 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:41.467182 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:41.467190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:41.467879 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:41.468689 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:41.468709 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:41.469923 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:41.469934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:41.471354 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:41.471365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:41.472144 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:41.472151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:41.472373 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:41.473018 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:41.473041 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:41.475332 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:41.478880 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:41.478888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:41.478904 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:41.479481 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:41.479913 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:41.479916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:41.479918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:41.480123 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:41.480786 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:41.480794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:41.481241 (+ 447us) client_negotiation.cc:769] Sending connection context
0925 11:06:41.481480 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 11:06:41.482083 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":285,"threads_started":1}
I20260925 11:06:41.484957 11813 heartbeater.cc:347] Connected to a master server at 127.8.181.62:42267
I20260925 11:06:41.485320 11813 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:41.486140 11813 heartbeater.cc:511] Master 127.8.181.62:42267 requested a full tablet report, sending...
I20260925 11:06:41.487970 11356 ts_manager.cc:194] Registered new tserver with Master: 2ec0b186316b443ca2d125f87879b8d2 (127.8.181.3:41737)
I20260925 11:06:41.488798 11356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:48005
I20260925 11:06:41.496033 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 11:06:41.529242 11681 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:06:41.526594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.8.181.62:42267)
0925 11:06:41.526841 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:41.526850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:41.527327 (+ 477us) server_negotiation.cc:407] Connection header received
0925 11:06:41.527476 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:41.527480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:41.527557 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:41.527666 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:41.528933 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55514: BlockingRecv error: recv got EOF from 127.0.0.1:55514 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:06:41.532254 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11417
W20260925 11:06:41.535962 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11417: Not authorized: Client connection negotiation failed: client connection to 127.8.181.1:41053: server requires authentication, but client does not have Kerberos credentials available
I20260925 11:06:41.536168 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11417
I20260925 11:06:41.549528 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11551
W20260925 11:06:41.552763 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11551: Not authorized: Client connection negotiation failed: client connection to 127.8.181.2:39591: server requires authentication, but client does not have Kerberos credentials available
I20260925 11:06:41.552932 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11551
I20260925 11:06:41.568708 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11685
W20260925 11:06:41.571928 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11685: Not authorized: Client connection negotiation failed: client connection to 127.8.181.3:41737: server requires authentication, but client does not have Kerberos credentials available
I20260925 11:06:41.572104 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11685
I20260925 11:06:41.587728 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11324
I20260925 11:06:41.589061 11386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:41.693722 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11324
2026-09-25T11:06:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3707 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:41 dist-test-slave-6042 krb5kdc[11830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:41 dist-test-slave-6042 krb5kdc[11830](info): set up 2 sockets
Sep 25 11:06:41 dist-test-slave-6042 krb5kdc[11830](info): commencing operation
krb5kdc: starting...
W20260925 11:06:42.808502 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:42 dist-test-slave-6042 krb5kdc[11830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:43.030881 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39833
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:33819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:39833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:43.363818 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:43.364223 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:43.364382 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:43.372189 11843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:43.372364 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:43.372480 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:43.372591 11843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:43.372671 11843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:43.382941 11843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:39833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:43.384346 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:43.385731 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:43.398330 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:43.398272 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:43.398296 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:43.399969 11843 server_base.cc:1034] running on GCE node
I20260925 11:06:43.400844 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:43.402412 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:43.403712 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790334403403681 us; error 42 us; skew 500 ppm
I20260925 11:06:43.404119 11843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:43 dist-test-slave-6042 krb5kdc[11830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334403, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:43.407604 11843 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:43.409281 11843 webserver.cc:460] Webserver started at http://127.8.181.62:44897/ using document root <none> and password file <none>
I20260925 11:06:43.410058 11843 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:43.410188 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:43.410611 11843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:43.413771 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "48282934290140fb8ec9c75f935ac400"
format_stamp: "Formatted at 2026-09-25 11:06:43 on dist-test-slave-6042"
server_key: "6c271cc0d762c267399f22aa11f4aca7"
server_key_iv: "b01652f532dfe6c320e3e88c26fb6fd9"
server_key_version: "encryptionkey@0"
I20260925 11:06:43.414690 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "48282934290140fb8ec9c75f935ac400"
format_stamp: "Formatted at 2026-09-25 11:06:43 on dist-test-slave-6042"
server_key: "6c271cc0d762c267399f22aa11f4aca7"
server_key_iv: "b01652f532dfe6c320e3e88c26fb6fd9"
server_key_version: "encryptionkey@0"
I20260925 11:06:43.420529 11843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:43.424377 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:43.425868 11843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:43.426082 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "48282934290140fb8ec9c75f935ac400"
format_stamp: "Formatted at 2026-09-25 11:06:43 on dist-test-slave-6042"
server_key: "6c271cc0d762c267399f22aa11f4aca7"
server_key_iv: "b01652f532dfe6c320e3e88c26fb6fd9"
server_key_version: "encryptionkey@0"
I20260925 11:06:43.426308 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:43.456759 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:43.461544 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:43.461875 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:43.481884 11843 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:39833
I20260925 11:06:43.481886 11909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:39833 every 8 connection(s)
I20260925 11:06:43.483337 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:43.487252 11910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:43.492074 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11843
I20260925 11:06:43.492290 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:43.493106 8916 external_mini_cluster.cc:1428] Setting key 460d36eafd48e84d13b508803bde868d
I20260925 11:06:43.499671 11910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400: Bootstrap starting.
I20260925 11:06:43.502978 11910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:43.504107 11910 log.cc:824] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:43.507012 11910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400: No bootstrap required, opened a new log
Sep 25 11:06:43 dist-test-slave-6042 krb5kdc[11830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:43.515596 11910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } }
I20260925 11:06:43.516044 11910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:43.516180 11910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48282934290140fb8ec9c75f935ac400, State: Initialized, Role: FOLLOWER
I20260925 11:06:43.516579 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:43.495205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.8.181.62:39833)
0925 11:06:43.496035 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:43.496047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:43.496089 (+ 42us) server_negotiation.cc:407] Connection header received
0925 11:06:43.498220 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:43.498246 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:43.498745 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:43.499151 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:43.500771 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:43.502117 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:43.503538 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:43.504264 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:43.504462 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:43.508561 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:43.508570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:43.508590 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:43.508630 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:43.511992 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:43.512765 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:43.512775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:43.512782 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:43.512949 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:43.513398 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:43.513407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:43.513410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:43.514021 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:43.514353 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:43.514852 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 11:06:43.515291 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":388,"threads_started":1}
I20260925 11:06:43.516909 11910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } }
I20260925 11:06:43.517226 11910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:43.517366 11910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:43.517552 11910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:43.520354 11910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } }
I20260925 11:06:43.520797 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:06:43.520998 11910 leader_election.cc:302] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48282934290140fb8ec9c75f935ac400; no voters:
I20260925 11:06:43.521615 11910 leader_election.cc:288] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:43.521735 11915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:43.523896 11910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:43.526782 11915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [term 1 LEADER]: Becoming Leader. State: Replica: 48282934290140fb8ec9c75f935ac400, State: Running, Role: LEADER
I20260925 11:06:43.527395 11915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } }
I20260925 11:06:43.530194 11916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48282934290140fb8ec9c75f935ac400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } } }
I20260925 11:06:43.530660 11916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:43.531167 11917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48282934290140fb8ec9c75f935ac400. Latest consensus state: current_term: 1 leader_uuid: "48282934290140fb8ec9c75f935ac400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48282934290140fb8ec9c75f935ac400" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39833 } } }
I20260925 11:06:43.531451 11917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:43.533962 11922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:43.539772 11922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:43.554638 11922 catalog_manager.cc:1347] Generated new cluster ID: 39736e325040469bbf904b2fcb230358
I20260925 11:06:43.554787 11922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:43.568914 11922 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:43.570943 11922 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:43.580438 11922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48282934290140fb8ec9c75f935ac400: Generated new TSK 0
I20260925 11:06:43.581633 11922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:06:43.583271 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:43.574669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.8.181.62:39833)
0925 11:06:43.574907 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:43.574917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:43.575522 (+ 605us) server_negotiation.cc:407] Connection header received
0925 11:06:43.575761 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:43.575766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:43.575831 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:43.575950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:43.576979 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:43.577740 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:43.579436 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:43.580045 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:43.580186 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:43.581590 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:43.581595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:43.581598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:43.582305 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:43.582373 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:43.582404 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:43.582590 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:43.582749 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 11:06:43.582853 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":73}
I20260925 11:06:43.590269 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11843
I20260925 11:06:43.591602 11905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:43.681540 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11843
2026-09-25T11:06:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1981 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:43 dist-test-slave-6042 krb5kdc[11945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:43 dist-test-slave-6042 krb5kdc[11945](info): set up 2 sockets
Sep 25 11:06:43 dist-test-slave-6042 krb5kdc[11945](info): commencing operation
krb5kdc: starting...
W20260925 11:06:44.783077 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:44 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:44.990604 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34749
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:45.316926 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:45.317231 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:45.317432 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:45.324767 11958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:45.324895 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:45.324941 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:45.324993 11958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:45.325037 11958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:45.335033 11958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:45.336417 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:45.337704 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:45.350337 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:45.350440 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:45.350384 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:45.351181 11958 server_base.cc:1034] running on GCE node
I20260925 11:06:45.352007 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:45.353507 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:45.354712 11958 hybrid_clock.cc:648] HybridClock initialized: now 1790334405354673 us; error 28 us; skew 500 ppm
I20260925 11:06:45.355068 11958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:45 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334405, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:45.358368 11958 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:45.359979 11958 webserver.cc:460] Webserver started at http://127.8.181.62:35529/ using document root <none> and password file <none>
I20260925 11:06:45.360703 11958 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:45.360852 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:45.361172 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:45.364319 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "97c8c3d68fbf43b290c7d916b5b2b76e"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "85866d59ff4028aa7cc5e13ad3b64192"
server_key_iv: "b9a2ee309b0dc37c07357ccb4c2f997b"
server_key_version: "encryptionkey@0"
I20260925 11:06:45.365088 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "97c8c3d68fbf43b290c7d916b5b2b76e"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "85866d59ff4028aa7cc5e13ad3b64192"
server_key_iv: "b9a2ee309b0dc37c07357ccb4c2f997b"
server_key_version: "encryptionkey@0"
I20260925 11:06:45.370425 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 11:06:45.374212 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:45.375825 11958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:45.376140 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "97c8c3d68fbf43b290c7d916b5b2b76e"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "85866d59ff4028aa7cc5e13ad3b64192"
server_key_iv: "b9a2ee309b0dc37c07357ccb4c2f997b"
server_key_version: "encryptionkey@0"
I20260925 11:06:45.376374 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:45.400671 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:45.404950 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:45.405267 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:45.424353 11958 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34749
I20260925 11:06:45.424399 12024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34749 every 8 connection(s)
I20260925 11:06:45.425778 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:45.429988 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:45.430737 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 11958
I20260925 11:06:45.430948 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:45.431613 8916 external_mini_cluster.cc:1428] Setting key afac4773d56a028056efcb10f99c6bb8
I20260925 11:06:45.444815 12025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e: Bootstrap starting.
Sep 25 11:06:45 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:45.448465 12025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:45.449944 12025 log.cc:824] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:45.453487 12025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e: No bootstrap required, opened a new log
I20260925 11:06:45.457401 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:45.433667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.8.181.62:34749)
0925 11:06:45.434486 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:45.434504 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:45.434554 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:06:45.436738 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:45.436769 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:45.437333 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:45.437793 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:45.439244 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:45.441270 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:45.442614 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:45.443680 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:45.443859 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:45.448257 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:45.448270 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:45.448293 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:45.448354 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:45.452272 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:45.453117 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:45.453125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:45.453133 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:45.453324 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:45.453845 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:45.453850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:45.453853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:45.454627 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:45.454953 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:45.455362 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 11:06:45.455858 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":318,"threads_started":1}
I20260925 11:06:45.462679 12025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } }
I20260925 11:06:45.463037 12025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:45.463114 12025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c8c3d68fbf43b290c7d916b5b2b76e, State: Initialized, Role: FOLLOWER
I20260925 11:06:45.463610 12025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } }
I20260925 11:06:45.463802 12025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:45.463887 12025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:45.464012 12025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:45.466132 12025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } }
I20260925 11:06:45.466728 12025 leader_election.cc:302] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c8c3d68fbf43b290c7d916b5b2b76e; no voters:
I20260925 11:06:45.467356 12025 leader_election.cc:288] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:45.467535 12030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:45.468287 12030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [term 1 LEADER]: Becoming Leader. State: Replica: 97c8c3d68fbf43b290c7d916b5b2b76e, State: Running, Role: LEADER
I20260925 11:06:45.468910 12030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } }
I20260925 11:06:45.469617 12025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:45.471827 12032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c8c3d68fbf43b290c7d916b5b2b76e. Latest consensus state: current_term: 1 leader_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } } }
I20260925 11:06:45.472198 12032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:45.472561 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c3d68fbf43b290c7d916b5b2b76e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34749 } } }
I20260925 11:06:45.472855 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:45.477910 12036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:45.483513 12036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:45.492961 12036 catalog_manager.cc:1347] Generated new cluster ID: b73fd372e3144fd289d338f6a79fccf9
I20260925 11:06:45.493093 12036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:45.516830 12036 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:45.518894 12036 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:45.532846 12036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c8c3d68fbf43b290c7d916b5b2b76e: Generated new TSK 0
I20260925 11:06:45.533809 12036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:45.598624 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:45.939568 12052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:45.939862 12052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:45.940019 12052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:45.947690 12052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:45.947820 12052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:45.948011 12052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:45.958618 12052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:45.959980 12052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:45.961406 12052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:45.975622 12057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:45.975559 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:45.975525 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:45.981995 12052 server_base.cc:1034] running on GCE node
I20260925 11:06:45.982810 12052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:45.983878 12052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:45.985126 12052 hybrid_clock.cc:648] HybridClock initialized: now 1790334405985070 us; error 93 us; skew 500 ppm
I20260925 11:06:45.985482 12052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:45 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334405, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:45.993960 12052 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:45.995803 12052 webserver.cc:460] Webserver started at http://127.8.181.1:43801/ using document root <none> and password file <none>
I20260925 11:06:45.996552 12052 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:45.996658 12052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:45.996942 12052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:46.000160 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "bb5db672191842cb81518e8ea6bed32d"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "7469c1169b2d9846c446fe3735a3c61f"
server_key_iv: "8d74280b98f68e5b313a059b3cb64d6b"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.000934 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "bb5db672191842cb81518e8ea6bed32d"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "7469c1169b2d9846c446fe3735a3c61f"
server_key_iv: "8d74280b98f68e5b313a059b3cb64d6b"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.006445 12052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 11:06:46.010317 12067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.012006 12052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:46.012266 12052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "bb5db672191842cb81518e8ea6bed32d"
format_stamp: "Formatted at 2026-09-25 11:06:45 on dist-test-slave-6042"
server_key: "7469c1169b2d9846c446fe3735a3c61f"
server_key_iv: "8d74280b98f68e5b313a059b3cb64d6b"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.012514 12052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:46.036703 12052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:46.041599 12052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:46.041956 12052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:46.043076 12052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:46.044526 12052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:46.044602 12052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.044739 12052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:46.044798 12052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.081099 12052 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:37463
I20260925 11:06:46.081193 12179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:37463 every 8 connection(s)
I20260925 11:06:46.082442 12052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:46.089262 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12052
I20260925 11:06:46.089483 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:46.090170 8916 external_mini_cluster.cc:1428] Setting key 5e43eb3cb107b26cee6cd41d1f89ec35
Sep 25 11:06:46 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334405, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:46.103101 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:46.085200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:34489 (local address 127.8.181.62:34749)
0925 11:06:46.085705 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:46.085710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:46.086570 (+ 860us) server_negotiation.cc:407] Connection header received
0925 11:06:46.088331 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:46.088337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:46.088442 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:46.088571 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:46.090223 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.091106 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.092706 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.093117 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.093266 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:46.097076 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:46.097082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:46.097085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:46.097120 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:46.099822 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:46.100643 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:46.100650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:46.100654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:46.100787 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:46.101235 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:46.101241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:46.101244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:46.101528 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:46.101692 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:46.102423 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 11:06:46.102720 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
I20260925 11:06:46.104161 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:46.085565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34749 (local address 127.8.181.1:34489)
0925 11:06:46.086290 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:46.086334 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:46.088049 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:46.088824 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:46.088833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:46.089389 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:46.090042 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:46.090061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.091274 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.091278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:46.092549 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:46.092561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.093370 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.093379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:46.093656 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:46.094399 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:46.094427 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:46.096808 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:46.099997 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:46.100014 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:46.100035 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:46.100479 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:46.100900 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:46.100907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:46.100910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:46.101119 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:46.101660 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:46.101669 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:46.102044 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 11:06:46.102332 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 11:06:46.102916 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":280,"threads_started":1}
I20260925 11:06:46.106048 12180 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34749
I20260925 11:06:46.106416 12180 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:46.107425 12180 heartbeater.cc:511] Master 127.8.181.62:34749 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:06:46.110031 11990 ts_manager.cc:194] Registered new tserver with Master: bb5db672191842cb81518e8ea6bed32d (127.8.181.1:37463)
I20260925 11:06:46.111660 11990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:34489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:46.138131 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:46.467505 12186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:46.467808 12186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:46.467901 12186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:46.475606 12186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:46.475744 12186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:46.475908 12186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:46.486498 12186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:46.487805 12186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:46.489086 12186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:46.502799 12194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:46.502861 12191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:46.503173 12192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:46.504009 12186 server_base.cc:1034] running on GCE node
I20260925 11:06:46.504611 12186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:46.505606 12186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:46.506807 12186 hybrid_clock.cc:648] HybridClock initialized: now 1790334406506760 us; error 36 us; skew 500 ppm
I20260925 11:06:46.507148 12186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:46 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334406, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:46.510601 12186 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:46.512327 12186 webserver.cc:460] Webserver started at http://127.8.181.2:37301/ using document root <none> and password file <none>
I20260925 11:06:46.513074 12186 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:46.513257 12186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:46.513635 12186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:46.516844 12186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "f158c9d2c5ec40e9ad92c82f577ccb10"
format_stamp: "Formatted at 2026-09-25 11:06:46 on dist-test-slave-6042"
server_key: "4d81af5c2e000294048a54e9033e8050"
server_key_iv: "b9f5c6833ed728d2045348250b02eaac"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.517935 12186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "f158c9d2c5ec40e9ad92c82f577ccb10"
format_stamp: "Formatted at 2026-09-25 11:06:46 on dist-test-slave-6042"
server_key: "4d81af5c2e000294048a54e9033e8050"
server_key_iv: "b9f5c6833ed728d2045348250b02eaac"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.523775 12186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 11:06:46.529289 12201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.531032 12186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:06:46.531179 12186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "f158c9d2c5ec40e9ad92c82f577ccb10"
format_stamp: "Formatted at 2026-09-25 11:06:46 on dist-test-slave-6042"
server_key: "4d81af5c2e000294048a54e9033e8050"
server_key_iv: "b9f5c6833ed728d2045348250b02eaac"
server_key_version: "encryptionkey@0"
I20260925 11:06:46.531363 12186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:46.555794 12186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:46.560846 12186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:46.561107 12186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:46.562044 12186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:46.563575 12186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:46.563650 12186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.563733 12186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:46.563778 12186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:46.599386 12186 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:39861
I20260925 11:06:46.599491 12313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:39861 every 8 connection(s)
I20260925 11:06:46.600839 12186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:46.609010 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12186
I20260925 11:06:46.609236 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:46.609838 8916 external_mini_cluster.cc:1428] Setting key 67ab8576042a28be2ea07ec32914aa7a
Sep 25 11:06:46 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334406, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:46.623965 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:46.603759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:34341 (local address 127.8.181.62:34749)
0925 11:06:46.604256 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:46.604260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:46.605325 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 11:06:46.607373 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:46.607378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:46.607452 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:46.607563 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:46.609585 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.610394 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.612396 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.612811 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.612959 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:46.616918 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:46.616925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:46.616930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:46.616982 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:46.620517 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:46.621404 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:46.621410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:46.621414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:46.621558 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:46.622024 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:46.622030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:46.622033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:46.622334 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:46.622514 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:46.623306 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 11:06:46.623573 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260925 11:06:46.625041 12317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:46.604215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34749 (local address 127.8.181.2:34341)
0925 11:06:46.605011 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:46.605158 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:46.607063 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:46.607809 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:46.607818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:46.608429 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:46.609367 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:46.609399 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.610597 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.610603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:46.612235 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:46.612247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:46.613082 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:46.613090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:46.613337 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:46.614127 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:46.614155 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:46.616564 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:46.620722 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:46.620733 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:46.620756 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:46.621250 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:46.621676 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:46.621682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:46.621686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:46.621899 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:46.622510 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:46.622521 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:46.622913 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 11:06:46.623207 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 11:06:46.623797 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":322,"threads_started":1}
I20260925 11:06:46.626744 12314 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34749
I20260925 11:06:46.627105 12314 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:46.627894 12314 heartbeater.cc:511] Master 127.8.181.62:34749 requested a full tablet report, sending...
I20260925 11:06:46.629544 11990 ts_manager.cc:194] Registered new tserver with Master: f158c9d2c5ec40e9ad92c82f577ccb10 (127.8.181.2:39861)
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:46.630548 11990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:34341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:46.678550 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:47.015846 12320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:47.016144 12320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:47.016237 12320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:47.024176 12320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:47.024302 12320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:47.024415 12320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:47.034585 12320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:47.036034 12320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:47.037326 12320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:47.050771 12326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:47.050729 12325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:47.050810 12328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:47.051677 12320 server_base.cc:1034] running on GCE node
I20260925 11:06:47.052217 12320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:47.053200 12320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:47.054525 12320 hybrid_clock.cc:648] HybridClock initialized: now 1790334407054417 us; error 39 us; skew 500 ppm
I20260925 11:06:47.054921 12320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:47.058183 12320 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:47.059829 12320 webserver.cc:460] Webserver started at http://127.8.181.3:33557/ using document root <none> and password file <none>
I20260925 11:06:47.060559 12320 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:47.060707 12320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:47.061021 12320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:47.064036 12320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "a18031120c9143e0b929de28a8d51d42"
format_stamp: "Formatted at 2026-09-25 11:06:47 on dist-test-slave-6042"
server_key: "0dfb91bc49a8a110d801ddee2ceabec9"
server_key_iv: "d904d3007bb40e5ea2c146a534569772"
server_key_version: "encryptionkey@0"
I20260925 11:06:47.064819 12320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "a18031120c9143e0b929de28a8d51d42"
format_stamp: "Formatted at 2026-09-25 11:06:47 on dist-test-slave-6042"
server_key: "0dfb91bc49a8a110d801ddee2ceabec9"
server_key_iv: "d904d3007bb40e5ea2c146a534569772"
server_key_version: "encryptionkey@0"
I20260925 11:06:47.070163 12320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 11:06:47.073825 12335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:47.075313 12320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:47.075491 12320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "a18031120c9143e0b929de28a8d51d42"
format_stamp: "Formatted at 2026-09-25 11:06:47 on dist-test-slave-6042"
server_key: "0dfb91bc49a8a110d801ddee2ceabec9"
server_key_iv: "d904d3007bb40e5ea2c146a534569772"
server_key_version: "encryptionkey@0"
I20260925 11:06:47.075697 12320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:47.100844 12320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:47.105069 12320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:47.105409 12320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:47.106372 12320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:47.107735 12320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:47.107838 12320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:47.107975 12320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:47.108044 12320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:47.114972 12180 heartbeater.cc:503] Master 127.8.181.62:34749 was elected leader, sending a full tablet report...
I20260925 11:06:47.143409 12320 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:36361
I20260925 11:06:47.143496 12447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:36361 every 8 connection(s)
I20260925 11:06:47.144738 12320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:47.149838 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12320
I20260925 11:06:47.150028 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:47.150625 8916 external_mini_cluster.cc:1428] Setting key 27d1bb9663828b3af22bf7c406c094e3
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:47.165235 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.147375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57123 (local address 127.8.181.62:34749)
0925 11:06:47.147884 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:47.147889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:47.148827 (+ 938us) server_negotiation.cc:407] Connection header received
0925 11:06:47.150907 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:47.150915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:47.151023 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:47.151145 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:47.152636 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.153452 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.155057 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.155456 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.155595 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.159134 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:47.159138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:47.159142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:47.159174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:47.161898 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.162771 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.162775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.162778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.162900 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.163351 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.163360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.163362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.163629 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:47.163809 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:47.164578 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 11:06:47.164819 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":244,"threads_started":1}
I20260925 11:06:47.166353 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.147928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34749 (local address 127.8.181.3:57123)
0925 11:06:47.148660 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:47.148709 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:47.150598 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:47.151391 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:47.151399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:47.151892 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:47.152466 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:47.152483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.153610 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.153616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:47.154904 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:47.154915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.155698 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.155706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:47.155934 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.156648 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:47.156671 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:47.158904 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:47.162050 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:47.162062 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:47.162081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:47.162636 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:47.163012 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:47.163018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:47.163021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:47.163230 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:47.163781 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:47.163790 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:47.164186 (+ 396us) client_negotiation.cc:769] Sending connection context
0925 11:06:47.164462 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 11:06:47.165059 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":278,"threads_started":1}
I20260925 11:06:47.167776 12448 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34749
I20260925 11:06:47.168097 12448 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:47.168828 12448 heartbeater.cc:511] Master 127.8.181.62:34749 requested a full tablet report, sending...
I20260925 11:06:47.170549 11990 ts_manager.cc:194] Registered new tserver with Master: a18031120c9143e0b929de28a8d51d42 (127.8.181.3:36361)
I20260925 11:06:47.171394 11990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:57123
I20260925 11:06:47.178122 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:47.231140 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.216747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.8.181.62:34749)
0925 11:06:47.216915 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:47.216920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:47.217472 (+ 552us) server_negotiation.cc:407] Connection header received
0925 11:06:47.217630 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:47.217634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:47.217751 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:47.217845 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:47.219623 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.220365 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.221585 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.221968 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.222059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.225849 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:47.225854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:47.225857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:47.225893 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:47.228561 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.229168 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.229171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.229174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.229331 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.229780 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.229783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.229785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.230104 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:47.230316 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:47.230732 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 11:06:47.230904 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 11:06:47.231822 11990 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45386
I20260925 11:06:47.234350 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12052
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:47.250646 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.235508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.8.181.1:37463)
0925 11:06:47.236111 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:47.236124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:47.236150 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:06:47.236206 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:47.236212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:47.236416 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:47.236542 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:47.237797 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.238923 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.240189 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.240755 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.240855 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.244919 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:47.244931 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:47.244945 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:47.244979 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:47.247902 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.248594 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.248601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.248603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.248728 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.249068 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.249071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.249073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.249401 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:47.249618 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:47.249889 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 11:06:47.250075 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":310,"threads_started":1}
W20260925 11:06:47.251926 12175 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38190
W20260925 11:06:47.253175 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12052: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:47.253394 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12052
I20260925 11:06:47.269604 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12186
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:47.286379 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.270628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42016 (local address 127.8.181.2:39861)
0925 11:06:47.271286 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:47.271293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:47.271322 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:06:47.271389 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:47.271397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:47.271628 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:47.271772 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:47.273017 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.274238 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.275614 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.276211 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.276319 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.280696 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:47.280718 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:47.280733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:47.280777 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:47.283736 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.284407 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.284415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.284417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.284558 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.284929 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.284932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.284934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.285273 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:47.285485 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:47.285774 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 11:06:47.285968 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
W20260925 11:06:47.287774 12309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42016
W20260925 11:06:47.289002 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12186: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:47.289158 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12186
I20260925 11:06:47.304914 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12320
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[11945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334407, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:47.322063 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:47.305949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.8.181.3:36361)
0925 11:06:47.306787 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:47.306794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:47.306823 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:06:47.306881 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:47.306887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:47.307099 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:47.307275 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:47.308630 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.309658 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.311082 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:47.311652 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:47.311794 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:47.316282 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:47.316295 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:47.316311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:47.316351 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:47.319622 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.320303 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.320310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.320313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.320428 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:47.320755 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:47.320759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:47.320760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:47.321089 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:47.321306 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:47.321592 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 11:06:47.321790 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":367,"threads_started":1}
W20260925 11:06:47.323402 12443 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56016
W20260925 11:06:47.324525 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12320: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:06:47.324702 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12320
I20260925 11:06:47.340135 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 11958
I20260925 11:06:47.341408 12020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:47.446177 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 11958
2026-09-25T11:06:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3770 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[12466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[12466](info): set up 2 sockets
Sep 25 11:06:47 dist-test-slave-6042 krb5kdc[12466](info): commencing operation
krb5kdc: starting...
W20260925 11:06:48.543267 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:48 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:48.762866 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34947
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:49.096505 12479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:49.096841 12479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:49.096931 12479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:49.104240 12479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:49.104367 12479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:49.104415 12479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:49.104465 12479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:49.104512 12479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:49.114748 12479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:49.116068 12479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:49.117336 12479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:49.129781 12484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:49.129819 12485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:49.129755 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:49.130721 12479 server_base.cc:1034] running on GCE node
I20260925 11:06:49.131592 12479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:49.133036 12479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:49.134275 12479 hybrid_clock.cc:648] HybridClock initialized: now 1790334409134223 us; error 47 us; skew 500 ppm
I20260925 11:06:49.134680 12479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:49 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334409, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:49.137861 12479 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:49.139458 12479 webserver.cc:460] Webserver started at http://127.8.181.62:36041/ using document root <none> and password file <none>
I20260925 11:06:49.140226 12479 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:49.140377 12479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:49.140702 12479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:49.143857 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "46d3ec805e4c46f394542576b6b6d264"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "98425069dd18378f7e00b9f855163541"
server_key_iv: "e3ce1ad17aafca1666901e8c397c37bd"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.144673 12479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "46d3ec805e4c46f394542576b6b6d264"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "98425069dd18378f7e00b9f855163541"
server_key_iv: "e3ce1ad17aafca1666901e8c397c37bd"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.149907 12479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 11:06:49.153565 12494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:49.155066 12479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:49.155309 12479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "46d3ec805e4c46f394542576b6b6d264"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "98425069dd18378f7e00b9f855163541"
server_key_iv: "e3ce1ad17aafca1666901e8c397c37bd"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.155524 12479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:49.177251 12479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:49.181901 12479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:49.182277 12479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:49.201840 12479 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34947
I20260925 11:06:49.201881 12545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34947 every 8 connection(s)
I20260925 11:06:49.203362 12479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:49.207185 12546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:49.213220 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12479
I20260925 11:06:49.213482 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:49.214284 8916 external_mini_cluster.cc:1428] Setting key b2687a43f7321da5542a93d27f3c1f6b
I20260925 11:06:49.218760 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264: Bootstrap starting.
I20260925 11:06:49.222704 12546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:49.223917 12546 log.cc:824] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:49.227198 12546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264: No bootstrap required, opened a new log
Sep 25 11:06:49 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:49.235730 12546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } }
I20260925 11:06:49.236193 12546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:49.236321 12546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d3ec805e4c46f394542576b6b6d264, State: Initialized, Role: FOLLOWER
I20260925 11:06:49.236981 12546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } }
I20260925 11:06:49.237250 12546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:49.237406 12546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:49.237587 12546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:49.239033 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:49.216554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.8.181.62:34947)
0925 11:06:49.217354 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:49.217366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:49.217407 (+ 41us) server_negotiation.cc:407] Connection header received
0925 11:06:49.219217 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:49.219241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:49.219640 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:49.219967 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:49.222046 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.223717 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.225309 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.225988 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.226127 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:49.230912 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:49.230920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:49.230936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:49.230974 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:49.234300 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:49.235156 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:49.235163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:49.235170 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:49.235341 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:49.235740 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:49.235745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:49.235755 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:49.236479 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:49.236818 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:49.237233 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 11:06:49.237884 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":392,"threads_started":1}
I20260925 11:06:49.240113 12546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } }
I20260925 11:06:49.240646 12546 leader_election.cc:302] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d3ec805e4c46f394542576b6b6d264; no voters:
I20260925 11:06:49.241346 12546 leader_election.cc:288] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:49.241501 12551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:49.242604 12551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [term 1 LEADER]: Becoming Leader. State: Replica: 46d3ec805e4c46f394542576b6b6d264, State: Running, Role: LEADER
I20260925 11:06:49.243217 12551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } }
I20260925 11:06:49.243743 12546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:49.246243 12552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d3ec805e4c46f394542576b6b6d264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } } }
I20260925 11:06:49.246268 12553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d3ec805e4c46f394542576b6b6d264. Latest consensus state: current_term: 1 leader_uuid: "46d3ec805e4c46f394542576b6b6d264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d3ec805e4c46f394542576b6b6d264" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34947 } } }
I20260925 11:06:49.246788 12553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:49.246802 12552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:49.250514 12560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:49.255877 12560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:49.264753 12560 catalog_manager.cc:1347] Generated new cluster ID: f342ce162bd44d9a962cf33a043ab705
I20260925 11:06:49.264881 12560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:49.276160 12560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:49.277792 12560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:49.284511 12560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d3ec805e4c46f394542576b6b6d264: Generated new TSK 0
I20260925 11:06:49.285385 12560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:49.340776 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:49.684409 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:49.684721 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:49.684901 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:49.692373 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:49.692503 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:49.692692 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:49.702598 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:49.703990 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:49.705305 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:49.718457 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:49.718431 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:49.718523 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:49.719467 12572 server_base.cc:1034] running on GCE node
I20260925 11:06:49.720124 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:49.721117 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:49.722394 12572 hybrid_clock.cc:648] HybridClock initialized: now 1790334409722348 us; error 78 us; skew 500 ppm
I20260925 11:06:49.722774 12572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:49 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334409, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:49.726328 12572 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:49.727972 12572 webserver.cc:460] Webserver started at http://127.8.181.1:45151/ using document root <none> and password file <none>
I20260925 11:06:49.728773 12572 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:49.728925 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:49.729242 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:49.732210 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "1104ed96666f4fe98898442a36151ba2"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "a03d1484ba43e2cf7fc08cb1c1f1a7a9"
server_key_iv: "6a6661156a67bebc97e0cc394800af1f"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.732996 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "1104ed96666f4fe98898442a36151ba2"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "a03d1484ba43e2cf7fc08cb1c1f1a7a9"
server_key_iv: "6a6661156a67bebc97e0cc394800af1f"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.738061 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:49.741596 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:49.743212 12572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:49.743439 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "1104ed96666f4fe98898442a36151ba2"
format_stamp: "Formatted at 2026-09-25 11:06:49 on dist-test-slave-6042"
server_key: "a03d1484ba43e2cf7fc08cb1c1f1a7a9"
server_key_iv: "6a6661156a67bebc97e0cc394800af1f"
server_key_version: "encryptionkey@0"
I20260925 11:06:49.743676 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:49.768823 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:49.772958 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:49.773284 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:49.774257 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:49.775699 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:49.775802 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:49.775961 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:49.776041 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:49.808890 12572 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33187
I20260925 11:06:49.808979 12699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33187 every 8 connection(s)
I20260925 11:06:49.810220 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:49.815266 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12572
I20260925 11:06:49.815488 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:49.816210 8916 external_mini_cluster.cc:1428] Setting key 8a173eae9069c8e555eaa69bebdb8d83
Sep 25 11:06:49 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334409, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:49.831130 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:49.812928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:55485 (local address 127.8.181.62:34947)
0925 11:06:49.813393 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:49.813397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:49.814161 (+ 764us) server_negotiation.cc:407] Connection header received
0925 11:06:49.816110 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:49.816116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:49.816217 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:49.816337 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:49.818263 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.819125 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.820696 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.821120 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.821263 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:49.825263 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:49.825268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:49.825272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:49.825304 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:49.827974 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:49.828701 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:49.828707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:49.828710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:49.828835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:49.829255 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:49.829261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:49.829264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:49.829546 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:49.829706 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:49.830432 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 11:06:49.830723 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":252,"threads_started":1}
I20260925 11:06:49.832098 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:49.813316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34947 (local address 127.8.181.1:55485)
0925 11:06:49.814006 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:49.814041 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:49.815839 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:49.816574 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:49.816582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:49.817242 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:49.818052 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:49.818073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.819294 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.819299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:49.820545 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:49.820557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:49.821350 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:49.821360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:49.821604 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:49.822322 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:49.822346 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:49.825038 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:49.828127 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:49.828137 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:49.828155 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:49.828577 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:49.828948 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:49.828953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:49.828957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:49.829143 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:49.829687 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:49.829695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:49.830059 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 11:06:49.830334 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 11:06:49.830916 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":277,"threads_started":1}
I20260925 11:06:49.833731 12700 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34947
I20260925 11:06:49.834051 12700 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:06:49.834899 12700 heartbeater.cc:511] Master 127.8.181.62:34947 requested a full tablet report, sending...
I20260925 11:06:49.837086 12511 ts_manager.cc:194] Registered new tserver with Master: 1104ed96666f4fe98898442a36151ba2 (127.8.181.1:33187)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:06:49.838618 12511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:55485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:49.862164 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:50.192912 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:50.193207 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:50.193300 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:50.200519 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:50.200698 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:50.200882 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:50.216571 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:50.218214 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:50.219964 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:50.235574 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:50.235476 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:50.235540 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:50.236291 12706 server_base.cc:1034] running on GCE node
I20260925 11:06:50.237134 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:50.238273 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:50.239493 12706 hybrid_clock.cc:648] HybridClock initialized: now 1790334410239386 us; error 102 us; skew 500 ppm
I20260925 11:06:50.239933 12706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:50.243534 12706 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:50.245453 12706 webserver.cc:460] Webserver started at http://127.8.181.2:45103/ using document root <none> and password file <none>
I20260925 11:06:50.246243 12706 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:50.246429 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:50.246862 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:50.250962 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "841d87e701dc4247b7a2b69a93a17113"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "d5ad465f60a0108f05291fcce6311acd"
server_key_iv: "f075abe7bc121417d414c9db702c7f60"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.251878 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "841d87e701dc4247b7a2b69a93a17113"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "d5ad465f60a0108f05291fcce6311acd"
server_key_iv: "f075abe7bc121417d414c9db702c7f60"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.257825 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 11:06:50.262152 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.263777 12706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:50.263978 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "841d87e701dc4247b7a2b69a93a17113"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "d5ad465f60a0108f05291fcce6311acd"
server_key_iv: "f075abe7bc121417d414c9db702c7f60"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.264210 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:50.280580 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:50.285567 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:50.285921 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:50.287099 12706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:50.288646 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:50.288767 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.288930 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:50.289016 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.326208 12706 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:39965
I20260925 11:06:50.326308 12833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:39965 every 8 connection(s)
I20260925 11:06:50.327622 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:50.333590 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12706
I20260925 11:06:50.333848 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:50.334352 8916 external_mini_cluster.cc:1428] Setting key ff876c754a8a3aa52f0335e6cc1b30e7
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:50.350404 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.330418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:49591 (local address 127.8.181.62:34947)
0925 11:06:50.330697 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:50.330704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:50.331815 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 11:06:50.333728 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:50.333735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:50.333839 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:50.333966 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:50.335961 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.336948 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.338612 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.339351 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.339497 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.343282 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:50.343288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:50.343291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:50.343324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:50.346322 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.347415 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.347421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.347423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.347643 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.348232 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.348237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.348239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.348691 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:50.348863 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:50.349853 (+ 990us) server_negotiation.cc:299] Negotiation successful
0925 11:06:50.350152 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 11:06:50.352079 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.330888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34947 (local address 127.8.181.2:49591)
0925 11:06:50.331639 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:50.331685 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:50.333426 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:50.334284 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:50.334292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:50.335002 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:50.335781 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:50.335804 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.337112 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.337116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:50.338374 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:50.338385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.339582 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.339587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:50.339808 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.340558 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:50.340580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:50.343041 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:50.346543 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:50.346556 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:50.346579 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:50.347276 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:50.347777 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:50.347786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:50.347792 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:50.348061 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:50.348867 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:50.348879 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:50.349406 (+ 527us) client_negotiation.cc:769] Sending connection context
0925 11:06:50.349735 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 11:06:50.350375 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":291,"threads_started":1}
I20260925 11:06:50.354023 12834 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34947
I20260925 11:06:50.354385 12834 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:50.355260 12834 heartbeater.cc:511] Master 127.8.181.62:34947 requested a full tablet report, sending...
I20260925 11:06:50.356950 12511 ts_manager.cc:194] Registered new tserver with Master: 841d87e701dc4247b7a2b69a93a17113 (127.8.181.2:39965)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:06:50.358132 12511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:49591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:50.386041 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:41609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:50.740839 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:50.741216 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:50.741394 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:50.749451 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:50.749615 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:50.749805 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:50.760382 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:34947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:50.761817 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:50.763228 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:50.778564 12844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:50.778607 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:50.778651 12847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:50.779479 12839 server_base.cc:1034] running on GCE node
I20260925 11:06:50.780140 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:50.781085 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:50.782291 12839 hybrid_clock.cc:648] HybridClock initialized: now 1790334410782245 us; error 36 us; skew 500 ppm
I20260925 11:06:50.782663 12839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:50.785948 12839 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:50.787637 12839 webserver.cc:460] Webserver started at http://127.8.181.3:39879/ using document root <none> and password file <none>
I20260925 11:06:50.788327 12839 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:50.788429 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:50.788779 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:50.791935 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "839d8fd2bc10458fa74ce459a8c3cbce"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "41d1387a722ca61603e6965483b18914"
server_key_iv: "cc42df512ef4647a4960109c11681c6f"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.792717 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "839d8fd2bc10458fa74ce459a8c3cbce"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "41d1387a722ca61603e6965483b18914"
server_key_iv: "cc42df512ef4647a4960109c11681c6f"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.798341 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:06:50.802304 12854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.803957 12839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:06:50.804168 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "839d8fd2bc10458fa74ce459a8c3cbce"
format_stamp: "Formatted at 2026-09-25 11:06:50 on dist-test-slave-6042"
server_key: "41d1387a722ca61603e6965483b18914"
server_key_iv: "cc42df512ef4647a4960109c11681c6f"
server_key_version: "encryptionkey@0"
I20260925 11:06:50.804409 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:50.828799 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:50.833775 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:50.834123 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:50.835212 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:50.836733 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:50.836848 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.837004 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:50.837077 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:50.841671 12700 heartbeater.cc:503] Master 127.8.181.62:34947 was elected leader, sending a full tablet report...
I20260925 11:06:50.870393 12839 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:41095
I20260925 11:06:50.870438 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:41095 every 8 connection(s)
I20260925 11:06:50.871686 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:50.877676 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 12839
I20260925 11:06:50.877911 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:50.878444 8916 external_mini_cluster.cc:1428] Setting key 6bfb125058068c3c29ccbc7ea99ba33e
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:50.893720 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.874541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:35177 (local address 127.8.181.62:34947)
0925 11:06:50.875019 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:50.875023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:50.875981 (+ 958us) server_negotiation.cc:407] Connection header received
0925 11:06:50.877853 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:50.877861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:50.877974 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:50.878109 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:50.879933 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.881198 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.883093 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.883583 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.883691 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.887352 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:50.887358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:50.887362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:50.887398 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:50.890254 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.891228 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.891234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.891236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.891365 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.891771 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.891775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.891777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.892078 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:50.892193 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:50.893012 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 11:06:50.893285 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260925 11:06:50.894688 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.875095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34947 (local address 127.8.181.3:35177)
0925 11:06:50.875800 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:50.875844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:50.877539 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:50.878346 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:50.878355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:50.879021 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:50.879759 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:50.879783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.881384 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.881388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:50.882913 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:50.882926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.883782 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.883786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:50.883983 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.884755 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:50.884778 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:50.887043 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:50.890438 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:50.890446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:50.890599 (+ 153us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:50.891070 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:50.891469 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:50.891472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:50.891474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:50.891654 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:50.892208 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:50.892214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:50.892548 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 11:06:50.892865 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 11:06:50.893329 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":270,"threads_started":1}
I20260925 11:06:50.896327 12967 heartbeater.cc:347] Connected to a master server at 127.8.181.62:34947
I20260925 11:06:50.896724 12967 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:50.897477 12967 heartbeater.cc:511] Master 127.8.181.62:34947 requested a full tablet report, sending...
I20260925 11:06:50.899075 12511 ts_manager.cc:194] Registered new tserver with Master: 839d8fd2bc10458fa74ce459a8c3cbce (127.8.181.3:41095)
I20260925 11:06:50.899865 12511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:35177
I20260925 11:06:50.907799 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:50.961794 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.944237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.8.181.1:33187)
0925 11:06:50.944997 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:50.945017 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:50.945046 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:06:50.945115 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:50.945123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:50.945357 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:50.945552 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:50.947356 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.948631 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.950035 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.950789 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.950903 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.955333 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:50.955350 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:50.955366 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:50.955408 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:50.958529 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.959538 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.959555 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.959558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.959782 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.960257 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.960261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.960264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.960723 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:50.960959 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:50.961299 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 11:06:50.961503 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":300,"threads_started":1}
Sep 25 11:06:50 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:50.981765 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.966718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.8.181.62:34947)
0925 11:06:50.966935 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:50.966942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:50.967310 (+ 368us) server_negotiation.cc:407] Connection header received
0925 11:06:50.967537 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:50.967541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:50.967628 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:50.967741 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:50.969157 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.969910 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.971539 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.971981 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.972094 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.976305 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:50.976310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:50.976314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:50.976353 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:50.979006 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.979810 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.979815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.979817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.979963 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:50.980382 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:50.980387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:50.980389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:50.980761 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:50.980976 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:50.981280 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 11:06:50.981459 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 11:06:50.982664 12511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42212
I20260925 11:06:50.983995 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12572
I20260925 11:06:51.006608 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:50.985016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50228 (local address 127.8.181.1:33187)
0925 11:06:50.985197 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:50.985202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:50.985597 (+ 395us) server_negotiation.cc:407] Connection header received
0925 11:06:50.985810 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:50.985814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:50.985921 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:50.986050 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:50.987538 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.988353 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.989896 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:50.990364 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:50.990586 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:50.993224 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:50.993230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:50.993233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:50.993273 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:51.003252 (+ 9979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.004368 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.004374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.004376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.004548 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.005056 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.005062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.005065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.005367 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:51.005665 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:51.006008 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 11:06:51.006280 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:06:51.007244 12695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:51.094977 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12572
I20260925 11:06:51.113571 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12706
Sep 25 11:06:51 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:51.131887 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:51.114638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.8.181.2:39965)
0925 11:06:51.115186 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:51.115193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:51.115220 (+ 27us) server_negotiation.cc:407] Connection header received
0925 11:06:51.115283 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:51.115290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:51.115516 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:51.115648 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:51.117007 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:51.118050 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:51.119472 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:51.120074 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:51.120190 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:51.125096 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:51.125113 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:51.125130 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:51.125194 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:51.128593 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.129438 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.129446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.129449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.129633 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.130057 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.130061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.130064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.130423 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:51.130765 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:51.131328 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 11:06:51.131545 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260925 11:06:51.133320 12829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:51.229249 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12706
I20260925 11:06:51.246255 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12839
Sep 25 11:06:51 dist-test-slave-6042 krb5kdc[12466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:51.265839 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:51.247354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.8.181.3:41095)
0925 11:06:51.248038 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:51.248045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:51.248146 (+ 101us) server_negotiation.cc:407] Connection header received
0925 11:06:51.248238 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:51.248245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:51.248479 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:51.248691 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:51.250194 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:51.251440 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:51.252840 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:51.253463 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:51.253926 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:51.259210 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:51.259235 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:51.259257 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:51.259320 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:51.262725 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.263584 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.263594 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.263597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.263739 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:51.264228 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:51.264232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:51.264234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:51.264691 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:51.264954 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:51.265258 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 11:06:51.265471 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":331,"threads_started":1}
I20260925 11:06:51.267280 12962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:51.353330 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12839
I20260925 11:06:51.370159 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 12479
I20260925 11:06:51.371780 12541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:51.482931 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 12479
2026-09-25T11:06:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4037 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:06:51 dist-test-slave-6042 krb5kdc[12987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:06:51 dist-test-slave-6042 krb5kdc[12987](info): set up 2 sockets
Sep 25 11:06:51 dist-test-slave-6042 krb5kdc[12987](info): commencing operation
krb5kdc: starting...
W20260925 11:06:52.603761 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.011s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:06:52 dist-test-slave-6042 krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:06:52.823556 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37799
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:53.200390 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:53.200727 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:53.200901 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:53.208536 13000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:53.208662 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:53.208707 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:53.208830 13000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:53.208932 13000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:53.219295 13000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:53.220719 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:53.222152 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:53.234613 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:53.234719 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:53.234642 13005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:53.240418 13000 server_base.cc:1034] running on GCE node
I20260925 11:06:53.241330 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:53.243000 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:53.244240 13000 hybrid_clock.cc:648] HybridClock initialized: now 1790334413244201 us; error 41 us; skew 500 ppm
I20260925 11:06:53.244666 13000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:53 dist-test-slave-6042 krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334413, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:53.248292 13000 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:06:53.249716 13000 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:53.249862 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:53.250216 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:53.253540 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "ec760c4cf02b4e0384f902e6e6d30b47"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "d2042d228c8b980969069d8c2cd0e704"
server_key_iv: "e2de5ab6a8b84869fdafe39f54004828"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.254547 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "ec760c4cf02b4e0384f902e6e6d30b47"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "d2042d228c8b980969069d8c2cd0e704"
server_key_iv: "e2de5ab6a8b84869fdafe39f54004828"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.260273 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:06:53.264212 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:53.265874 13000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 11:06:53.266091 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "ec760c4cf02b4e0384f902e6e6d30b47"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "d2042d228c8b980969069d8c2cd0e704"
server_key_iv: "e2de5ab6a8b84869fdafe39f54004828"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.266346 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:53.292820 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:53.297724 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:53.298028 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:53.319284 13000 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:37799
I20260925 11:06:53.319301 13065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:37799 every 8 connection(s)
I20260925 11:06:53.320287 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:53.324293 13066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:53.327281 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13000
I20260925 11:06:53.327579 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:53.328361 8916 external_mini_cluster.cc:1428] Setting key f82e0708a6a1b223432cb7a606facd2e
I20260925 11:06:53.338325 13066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: Bootstrap starting.
I20260925 11:06:53.342548 13066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:53.344002 13066 log.cc:824] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:06:53 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:53.347350 13066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: No bootstrap required, opened a new log
I20260925 11:06:53.353462 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:53.330780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.8.181.62:37799)
0925 11:06:53.331764 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:53.331777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:53.331821 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:06:53.333688 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:53.333725 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:53.334186 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:53.334681 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:53.336552 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.337859 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.339640 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.340361 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.340512 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:53.345829 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:53.345839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:53.345856 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:53.345893 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:53.349022 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:53.349815 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:53.349822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:53.349828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:53.350009 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:53.350442 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:53.350446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:53.350448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:53.351120 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:53.351470 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:53.351893 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 11:06:53.352350 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":476,"threads_started":1}
I20260925 11:06:53.355784 13066 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } }
I20260925 11:06:53.356187 13066 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:53.356315 13066 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec760c4cf02b4e0384f902e6e6d30b47, State: Initialized, Role: FOLLOWER
I20260925 11:06:53.356976 13066 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } }
I20260925 11:06:53.357244 13066 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:53.357386 13066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:53.357558 13066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:53.360476 13066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } }
I20260925 11:06:53.361035 13066 leader_election.cc:302] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec760c4cf02b4e0384f902e6e6d30b47; no voters:
I20260925 11:06:53.361694 13066 leader_election.cc:288] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:53.361864 13071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:53.362690 13071 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [term 1 LEADER]: Becoming Leader. State: Replica: ec760c4cf02b4e0384f902e6e6d30b47, State: Running, Role: LEADER
I20260925 11:06:53.363729 13071 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } }
I20260925 11:06:53.364300 13066 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:53.367120 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } } }
I20260925 11:06:53.367097 13073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec760c4cf02b4e0384f902e6e6d30b47. Latest consensus state: current_term: 1 leader_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec760c4cf02b4e0384f902e6e6d30b47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37799 } } }
I20260925 11:06:53.367584 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:53.367630 13073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47 [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:53.368631 13078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:53.378105 13078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:53.388254 13078 catalog_manager.cc:1347] Generated new cluster ID: 3781452d8a3c45b19293f341d15b9c55
I20260925 11:06:53.388391 13078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:53.412741 13078 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:53.414337 13078 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:53.422765 13078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: Generated new TSK 0
I20260925 11:06:53.423748 13078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:06:53.488193 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:53.845220 13092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:53.845547 13092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:53.845697 13092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:53.853539 13092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:53.853672 13092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:53.853873 13092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:53.865377 13092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:53.867076 13092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:53.868942 13092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:53.883901 13097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:53.883939 13100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:53.883961 13098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:53.884824 13092 server_base.cc:1034] running on GCE node
I20260925 11:06:53.885457 13092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:53.886655 13092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:53.887919 13092 hybrid_clock.cc:648] HybridClock initialized: now 1790334413887856 us; error 58 us; skew 500 ppm
I20260925 11:06:53.888315 13092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:53 dist-test-slave-6042 krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334413, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:53.891917 13092 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:06:53.893199 13092 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:53.893340 13092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:53.893666 13092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:53.896840 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "ed5e689480db43ee82889a67b91a3dca"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "8086e6c4cc706fdd2bcef169f3b7ec1d"
server_key_iv: "03198578dbdb69b72bdaafc1d4b147b9"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.897728 13092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "ed5e689480db43ee82889a67b91a3dca"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "8086e6c4cc706fdd2bcef169f3b7ec1d"
server_key_iv: "03198578dbdb69b72bdaafc1d4b147b9"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.903278 13092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 11:06:53.907406 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:53.909111 13092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260925 11:06:53.909360 13092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "ed5e689480db43ee82889a67b91a3dca"
format_stamp: "Formatted at 2026-09-25 11:06:53 on dist-test-slave-6042"
server_key: "8086e6c4cc706fdd2bcef169f3b7ec1d"
server_key_iv: "03198578dbdb69b72bdaafc1d4b147b9"
server_key_version: "encryptionkey@0"
I20260925 11:06:53.909634 13092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:53.934386 13092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:53.939998 13092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:53.940311 13092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:53.941372 13092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:53.942891 13092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:53.943020 13092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:53.943171 13092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:53.943264 13092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:53.980127 13092 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34669
I20260925 11:06:53.980219 13218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34669 every 8 connection(s)
I20260925 11:06:53.981071 13092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:53.981926 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13092
I20260925 11:06:53.982142 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:53.982925 8916 external_mini_cluster.cc:1428] Setting key aaaccceee65a45f701e4db43d99dc637
Sep 25 11:06:53 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334413, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:06:54.006403 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:53.983798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:39975 (local address 127.8.181.62:37799)
0925 11:06:53.984284 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:53.984289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:53.985558 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 11:06:53.988697 (+ 3139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:53.988702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:53.988780 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:53.988883 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:53.991281 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.992180 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.993890 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.994720 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.994871 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:53.998623 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:53.998633 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:53.998640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:53.998690 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:54.002638 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:54.003596 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:54.003604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:54.003609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:54.003787 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:54.004316 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:54.004324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:54.004328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:54.004743 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:54.004928 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:54.005833 (+ 905us) server_negotiation.cc:299] Negotiation successful
0925 11:06:54.006049 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260925 11:06:54.007475 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:53.984387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37799 (local address 127.8.181.1:39975)
0925 11:06:53.985379 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:53.985425 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:53.988252 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:53.989207 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:53.989222 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:53.990091 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:53.991045 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:53.991076 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.992413 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.992417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:53.993702 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:53.993715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:53.994953 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:53.994958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:53.995175 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:53.995937 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:53.995961 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:53.998304 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:54.002855 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:54.002864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:54.002881 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:54.003432 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:54.003958 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:54.003962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:54.003965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:54.004170 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:54.004896 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:54.004902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:54.005318 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 11:06:54.005697 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 11:06:54.006140 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":345,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:06:54.009457 13219 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37799
I20260925 11:06:54.009815 13219 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:54.010708 13219 heartbeater.cc:511] Master 127.8.181.62:37799 requested a full tablet report, sending...
I20260925 11:06:54.013214 13031 ts_manager.cc:194] Registered new tserver with Master: ed5e689480db43ee82889a67b91a3dca (127.8.181.1:34669)
I20260925 11:06:54.014945 13031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:39975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:06:54.035270 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:54.375900 13225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:54.376204 13225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:54.376303 13225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:54.383819 13225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:54.383948 13225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:54.384068 13225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:54.394163 13225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:54.395586 13225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:54.396872 13225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:54.410027 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:54.409981 13233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:54.409999 13231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:54.410734 13225 server_base.cc:1034] running on GCE node
I20260925 11:06:54.411348 13225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:54.412294 13225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:54.413491 13225 hybrid_clock.cc:648] HybridClock initialized: now 1790334414413447 us; error 42 us; skew 500 ppm
I20260925 11:06:54.413856 13225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:54 dist-test-slave-6042 krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334414, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:54.417147 13225 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:06:54.418424 13225 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:54.418602 13225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:54.418938 13225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:54.421998 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "daa5d5b674de43509944fcfed11dcb95"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "6c4b2006364db1ef17bb002ac7522138"
server_key_iv: "2e69af7ba191bb51031d7ab1516d879a"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.422937 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "daa5d5b674de43509944fcfed11dcb95"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "6c4b2006364db1ef17bb002ac7522138"
server_key_iv: "2e69af7ba191bb51031d7ab1516d879a"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.428529 13225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 11:06:54.432543 13239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:54.434151 13225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 11:06:54.434367 13225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "daa5d5b674de43509944fcfed11dcb95"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "6c4b2006364db1ef17bb002ac7522138"
server_key_iv: "2e69af7ba191bb51031d7ab1516d879a"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.434715 13225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:54.473021 13225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:54.477728 13225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:54.478031 13225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:54.479282 13225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:54.480907 13225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:54.481074 13225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:54.481227 13225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:54.481333 13225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:54.520046 13225 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:41721
I20260925 11:06:54.520081 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:41721 every 8 connection(s)
I20260925 11:06:54.521019 13225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:54.527786 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13225
I20260925 11:06:54.528038 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:54.528671 8916 external_mini_cluster.cc:1428] Setting key 46610a2c1c679bc53d912a00ed780b12
Sep 25 11:06:54 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334414, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:54.549237 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:54.523918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:39365 (local address 127.8.181.62:37799)
0925 11:06:54.524597 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:54.524604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:54.525719 (+ 1115us) server_negotiation.cc:407] Connection header received
0925 11:06:54.528951 (+ 3232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:54.528960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:54.529093 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:54.529234 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:54.532049 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:54.533299 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:54.536088 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:54.536621 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:54.536756 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:54.541638 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:54.541644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:54.541648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:54.541700 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:54.545116 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:54.546327 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:54.546332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:54.546334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:54.546544 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:54.547208 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:54.547211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:54.547213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:54.547480 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:54.547593 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:54.548769 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0925 11:06:54.548959 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":369,"threads_started":1}
I20260925 11:06:54.551072 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:54.524387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37799 (local address 127.8.181.2:39365)
0925 11:06:54.525462 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:54.525542 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:54.528484 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:54.529797 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:54.529839 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:54.530751 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:54.531757 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:54.531818 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:54.533510 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:54.533518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:54.535858 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:54.535884 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:54.536892 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:54.536901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:54.537235 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:54.538137 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:54.538161 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:54.541276 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:54.545321 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:54.545337 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:54.545364 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:54.546146 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:54.546755 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:54.546765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:54.546771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:54.547063 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:54.547630 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:54.547644 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:54.548236 (+ 592us) client_negotiation.cc:769] Sending connection context
0925 11:06:54.548635 (+ 399us) client_negotiation.cc:240] Negotiation successful
0925 11:06:54.549320 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":403,"threads_started":1}
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:06:54.552915 13352 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37799
I20260925 11:06:54.553383 13352 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:54.554128 13352 heartbeater.cc:511] Master 127.8.181.62:37799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:06:54.555908 13031 ts_manager.cc:194] Registered new tserver with Master: daa5d5b674de43509944fcfed11dcb95 (127.8.181.2:41721)
I20260925 11:06:54.556834 13031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:39365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:06:54.583977 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:06:54.929183 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:54.929493 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:54.929675 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:54.937731 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:54.937858 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:54.938040 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:54.948482 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:54.949888 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:54.951375 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:54.969849 13363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:54.969916 13366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:54.969969 13364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:54.971009 13358 server_base.cc:1034] running on GCE node
I20260925 11:06:54.971692 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:54.972886 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:54.974129 13358 hybrid_clock.cc:648] HybridClock initialized: now 1790334414974079 us; error 43 us; skew 500 ppm
I20260925 11:06:54.974659 13358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:06:54 dist-test-slave-6042 krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334414, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:06:54.978242 13358 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:06:54.979677 13358 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:54.979808 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:54.980176 13358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:54.984385 13358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "ec3ba9fedcc04fcea84dcfed01af273e"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "bfc28d107e7d0de93a0c442cdae4dc5a"
server_key_iv: "e14aaeb040ad2411e9c149106dbada04"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.985555 13358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "ec3ba9fedcc04fcea84dcfed01af273e"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "bfc28d107e7d0de93a0c442cdae4dc5a"
server_key_iv: "e14aaeb040ad2411e9c149106dbada04"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.991312 13358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 11:06:54.995560 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:54.997294 13358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260925 11:06:54.997519 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "ec3ba9fedcc04fcea84dcfed01af273e"
format_stamp: "Formatted at 2026-09-25 11:06:54 on dist-test-slave-6042"
server_key: "bfc28d107e7d0de93a0c442cdae4dc5a"
server_key_iv: "e14aaeb040ad2411e9c149106dbada04"
server_key_version: "encryptionkey@0"
I20260925 11:06:54.997794 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:55.018198 13219 heartbeater.cc:503] Master 127.8.181.62:37799 was elected leader, sending a full tablet report...
I20260925 11:06:55.034253 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:55.039136 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:55.039378 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:55.040417 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:55.041746 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:55.041828 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:55.041904 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:55.041957 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:55.077829 13358 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45539
I20260925 11:06:55.077935 13484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45539 every 8 connection(s)
I20260925 11:06:55.078765 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:55.087839 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13358
I20260925 11:06:55.088047 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:55.088590 8916 external_mini_cluster.cc:1428] Setting key 95e8a73a545727c310266e06f0cef670
Sep 25 11:06:55 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334414, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:06:55.101030 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.081487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:60649 (local address 127.8.181.62:37799)
0925 11:06:55.082018 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.082023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.083215 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 11:06:55.085099 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.085104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.085195 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.085325 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:55.087131 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.088372 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.090402 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.091067 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.091219 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.095171 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:55.095178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:55.095183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:55.095234 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:55.097917 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.098871 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.098876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.098878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.098998 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.099379 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.099382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.099384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.099645 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:55.099768 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.100546 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.100738 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
I20260925 11:06:55.102136 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.082084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37799 (local address 127.8.181.3:60649)
0925 11:06:55.083019 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.083071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.084766 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.085575 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.085584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.086142 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:55.086919 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.086941 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.088557 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.088561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.090238 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.090251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.091317 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.091324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.091594 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.092457 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:06:55.092483 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:55.094911 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:55.098083 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:55.098091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:55.098106 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:55.098739 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:55.099094 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:06:55.099097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:06:55.099099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:06:55.099280 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:06:55.099804 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:55.099811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:55.100153 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.100412 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.100868 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":509,"threads_started":1}
I20260925 11:06:55.104067 13485 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37799
I20260925 11:06:55.104420 13485 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:55.105088 13485 heartbeater.cc:511] Master 127.8.181.62:37799 requested a full tablet report, sending...
I20260925 11:06:55.106639 13030 ts_manager.cc:194] Registered new tserver with Master: ec3ba9fedcc04fcea84dcfed01af273e (127.8.181.3:45539)
I20260925 11:06:55.107604 13030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:60649
I20260925 11:06:55.117524 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:06:55.136312 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.122700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.8.181.62:37799)
0925 11:06:55.122942 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.122948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.123216 (+ 268us) server_negotiation.cc:407] Connection header received
0925 11:06:55.123355 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.123364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.123437 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.123617 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:55.125253 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.126016 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.127406 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.127828 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.127955 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.130631 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:55.130639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:55.130643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:55.130678 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:55.133284 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.134053 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.134059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.134062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.134208 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.134707 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.134712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.134714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.135094 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:55.135311 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.135754 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.136019 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:06:55.141271 13030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:55.148361 13030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:55.188974 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.161948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.8.181.1:34669)
0925 11:06:55.178423 (+ 16475us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.178435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.179679 (+ 1244us) server_negotiation.cc:407] Connection header received
0925 11:06:55.180244 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.180253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.180516 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.180695 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:55.182095 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.183328 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.186088 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.187404 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.187537 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.187792 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.188440 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.188640 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16256,"thread_start_us":15949,"threads_started":1}
I20260925 11:06:55.189073 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.161615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34669 (local address 127.0.0.1:35840)
0925 11:06:55.179556 (+ 17941us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.179578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.179999 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.181058 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.181062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.181097 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:55.181839 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.181857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.183541 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.183546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.185902 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.185921 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.187950 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.187956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.188160 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.188215 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.188369 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.188599 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17598,"mutex_wait_us":227,"spinlock_wait_cycles":306176,"thread_start_us":17534,"threads_started":1}
I20260925 11:06:55.189987 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.161299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45539 (local address 127.0.0.1:57214)
0925 11:06:55.179171 (+ 17872us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.179226 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.179956 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.181051 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.181059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.181105 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:55.181833 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.181845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.184005 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.184011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.186775 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.186791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.188899 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.188906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.189358 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.189392 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.189490 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.189631 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17479,"mutex_wait_us":261,"spinlock_wait_cycles":108672,"thread_start_us":17298,"threads_started":1}
I20260925 11:06:55.190078 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.161545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.8.181.3:45539)
0925 11:06:55.177448 (+ 15903us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.177463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.179378 (+ 1915us) server_negotiation.cc:407] Connection header received
0925 11:06:55.180262 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.180268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.180521 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.180694 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:55.182083 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.183838 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.186940 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.188437 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.188554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.188755 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.189555 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.189753 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15628,"thread_start_us":15941,"threads_started":1}
I20260925 11:06:55.190213 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.163095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.8.181.2:41721)
0925 11:06:55.171013 (+ 7918us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.171028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.180157 (+ 9129us) server_negotiation.cc:407] Connection header received
0925 11:06:55.180509 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.180517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.180756 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.180945 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:55.182157 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.183821 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.187317 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.188817 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.188917 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.189130 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.189718 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.189877 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7667,"thread_start_us":7487,"threads_started":1}
I20260925 11:06:55.190999 13500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.162827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:41721 (local address 127.0.0.1:43776)
0925 11:06:55.179990 (+ 17163us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.180009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.180318 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.181263 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.181272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.181301 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:55.181961 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.181975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.184007 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.184011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.186775 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.186791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.189329 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.189336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.189520 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.189552 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.190382 (+ 830us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.190612 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16835,"mutex_wait_us":94,"thread_start_us":15889,"threads_started":1}
I20260925 11:06:55.193768 13287 tablet_service.cc:1467] Processing CreateTablet for tablet 18efe3f13d054251890bcdf0b227513f (DEFAULT_TABLE table=test-table [id=caab2d6ca74e471a9d4e175c786a46e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:55.193533 13154 tablet_service.cc:1467] Processing CreateTablet for tablet 18efe3f13d054251890bcdf0b227513f (DEFAULT_TABLE table=test-table [id=caab2d6ca74e471a9d4e175c786a46e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:55.194154 13420 tablet_service.cc:1467] Processing CreateTablet for tablet 18efe3f13d054251890bcdf0b227513f (DEFAULT_TABLE table=test-table [id=caab2d6ca74e471a9d4e175c786a46e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:55.195065 13287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18efe3f13d054251890bcdf0b227513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:55.195120 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18efe3f13d054251890bcdf0b227513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:55.195469 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18efe3f13d054251890bcdf0b227513f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:55.207795 13504 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Bootstrap starting.
I20260925 11:06:55.210167 13506 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Bootstrap starting.
I20260925 11:06:55.211037 13505 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Bootstrap starting.
I20260925 11:06:55.213048 13504 tablet_bootstrap.cc:654] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:55.214246 13504 log.cc:824] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:55.214839 13505 tablet_bootstrap.cc:654] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:55.214839 13506 tablet_bootstrap.cc:654] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:55.216228 13506 log.cc:824] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:55.216228 13505 log.cc:824] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:55.217329 13504 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: No bootstrap required, opened a new log
I20260925 11:06:55.217656 13504 ts_tablet_manager.cc:1397] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 11:06:55.219560 13505 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: No bootstrap required, opened a new log
I20260925 11:06:55.219964 13505 ts_tablet_manager.cc:1397] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 11:06:55.220868 13506 tablet_bootstrap.cc:492] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: No bootstrap required, opened a new log
I20260925 11:06:55.221149 13506 ts_tablet_manager.cc:1397] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260925 11:06:55.225692 13504 raft_consensus.cc:348] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.226203 13504 raft_consensus.cc:374] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:55.226336 13504 raft_consensus.cc:729] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa5d5b674de43509944fcfed11dcb95, State: Initialized, Role: FOLLOWER
I20260925 11:06:55.227051 13504 consensus_queue.cc:260] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.230072 13352 heartbeater.cc:503] Master 127.8.181.62:37799 was elected leader, sending a full tablet report...
I20260925 11:06:55.230934 13504 ts_tablet_manager.cc:1428] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 11:06:55.231155 13505 raft_consensus.cc:348] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.231648 13505 raft_consensus.cc:374] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:55.231818 13505 raft_consensus.cc:729] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5e689480db43ee82889a67b91a3dca, State: Initialized, Role: FOLLOWER
I20260925 11:06:55.232125 13506 raft_consensus.cc:348] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.232632 13506 raft_consensus.cc:374] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:55.232537 13505 consensus_queue.cc:260] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.232771 13506 raft_consensus.cc:729] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3ba9fedcc04fcea84dcfed01af273e, State: Initialized, Role: FOLLOWER
I20260925 11:06:55.233490 13506 consensus_queue.cc:260] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.235613 13485 heartbeater.cc:503] Master 127.8.181.62:37799 was elected leader, sending a full tablet report...
I20260925 11:06:55.235746 13506 ts_tablet_manager.cc:1428] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260925 11:06:55.237900 13505 ts_tablet_manager.cc:1428] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20260925 11:06:55.267810 13512 raft_consensus.cc:482] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:55.268187 13512 raft_consensus.cc:504] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.270679 13512 leader_election.cc:288] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed5e689480db43ee82889a67b91a3dca (127.8.181.1:34669), daa5d5b674de43509944fcfed11dcb95 (127.8.181.2:41721)
W20260925 11:06:55.273073 13353 tablet.cc:2369] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:55.277230 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.270941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:41721 (local address 127.8.181.3:37091)
0925 11:06:55.271246 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.271270 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.271401 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.271815 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.271820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.271848 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:55.272066 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.272080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.273488 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.273492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.275310 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.275326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.276628 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.276633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.276778 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.276803 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.276888 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.276993 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":24}
I20260925 11:06:55.277333 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.271161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:37091 (local address 127.8.181.2:41721)
0925 11:06:55.271382 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.271390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.271415 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:06:55.271613 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.271618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.271678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.271792 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:55.272239 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.273318 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.275490 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.276431 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.276583 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.276692 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.276941 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.277080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:06:55.278319 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.271158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34669 (local address 127.8.181.3:55065)
0925 11:06:55.271945 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:55.271960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:55.272068 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:55.272374 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:55.272378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:55.272403 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:06:55.272857 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.272869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.274303 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.274308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.276034 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:55.276045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.277734 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.277741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:55.277881 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.277904 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 11:06:55.277979 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 11:06:55.278076 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":1,"thread_start_us":485,"threads_started":1}
I20260925 11:06:55.278402 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.271354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55065 (local address 127.8.181.1:34669)
0925 11:06:55.271556 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.271562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.272058 (+ 496us) server_negotiation.cc:407] Connection header received
0925 11:06:55.272176 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.272182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.272256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.272321 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:06:55.273042 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.274167 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.276231 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.277380 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.277517 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.277625 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.278037 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.278171 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:06:55.278402 13307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18efe3f13d054251890bcdf0b227513f" candidate_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa5d5b674de43509944fcfed11dcb95" is_pre_election: true
I20260925 11:06:55.279043 13307 raft_consensus.cc:2454] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3ba9fedcc04fcea84dcfed01af273e in term 0.
I20260925 11:06:55.279354 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18efe3f13d054251890bcdf0b227513f" candidate_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5e689480db43ee82889a67b91a3dca" is_pre_election: true
I20260925 11:06:55.279788 13373 leader_election.cc:302] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: daa5d5b674de43509944fcfed11dcb95, ec3ba9fedcc04fcea84dcfed01af273e; no voters:
I20260925 11:06:55.279902 13174 raft_consensus.cc:2454] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3ba9fedcc04fcea84dcfed01af273e in term 0.
I20260925 11:06:55.280252 13512 raft_consensus.cc:2781] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:55.280429 13512 raft_consensus.cc:482] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:55.280545 13512 raft_consensus.cc:3037] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:55.282795 13512 raft_consensus.cc:504] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.283450 13512 leader_election.cc:288] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [CANDIDATE]: Term 1 election: Requested vote from peers ed5e689480db43ee82889a67b91a3dca (127.8.181.1:34669), daa5d5b674de43509944fcfed11dcb95 (127.8.181.2:41721)
I20260925 11:06:55.284152 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18efe3f13d054251890bcdf0b227513f" candidate_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5e689480db43ee82889a67b91a3dca"
I20260925 11:06:55.284399 13174 raft_consensus.cc:3037] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:55.284380 13307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18efe3f13d054251890bcdf0b227513f" candidate_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daa5d5b674de43509944fcfed11dcb95"
I20260925 11:06:55.284708 13307 raft_consensus.cc:3037] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:55.286818 13174 raft_consensus.cc:2454] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3ba9fedcc04fcea84dcfed01af273e in term 1.
I20260925 11:06:55.286988 13307 raft_consensus.cc:2454] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3ba9fedcc04fcea84dcfed01af273e in term 1.
I20260925 11:06:55.287333 13376 leader_election.cc:302] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec3ba9fedcc04fcea84dcfed01af273e, ed5e689480db43ee82889a67b91a3dca; no voters:
I20260925 11:06:55.287704 13512 raft_consensus.cc:2781] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:55.288151 13512 raft_consensus.cc:686] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 1 LEADER]: Becoming Leader. State: Replica: ec3ba9fedcc04fcea84dcfed01af273e, State: Running, Role: LEADER
I20260925 11:06:55.288774 13512 consensus_queue.cc:237] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } }
I20260925 11:06:55.293859 13030 catalog_manager.cc:5495] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3ba9fedcc04fcea84dcfed01af273e (127.8.181.3). New cstate: current_term: 1 leader_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ba9fedcc04fcea84dcfed01af273e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 } health_report { overall_health: UNKNOWN } } }
I20260925 11:06:55.328125 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.321596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.8.181.3:45539)
0925 11:06:55.321855 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.321861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.321911 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:06:55.322143 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.322148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.322261 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.322440 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:06:55.322919 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.324036 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.325462 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.326082 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.326251 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.326804 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:06:55.327494 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:06:55.327636 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.327761 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.327851 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260925 11:06:55.330327 13486 tablet.cc:2369] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:55.337769 13174 raft_consensus.cc:1261] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 1 FOLLOWER]: Refusing update from remote peer ec3ba9fedcc04fcea84dcfed01af273e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:55.337733 13307 raft_consensus.cc:1261] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 1 FOLLOWER]: Refusing update from remote peer ec3ba9fedcc04fcea84dcfed01af273e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:55.338980 13512 consensus_queue.cc:1035] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5e689480db43ee82889a67b91a3dca" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:55.339318 13514 consensus_queue.cc:1035] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [LEADER]: Connected to new peer: Peer: permanent_uuid: "daa5d5b674de43509944fcfed11dcb95" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:55.351235 13517 mvcc.cc:205] Tried to move back new op lower bound from 7333209765212094464 to 7333209765027225600. Current Snapshot: MvccSnapshot[applied={T|T < 7333209765212094464}]
I20260925 11:06:55.369589 13030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:55.369886 13030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:55.373759 13030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec760c4cf02b4e0384f902e6e6d30b47: Sending DeleteTablet for 3 replicas of tablet 18efe3f13d054251890bcdf0b227513f
I20260925 11:06:55.375072 13287 tablet_service.cc:1514] Processing DeleteTablet for tablet 18efe3f13d054251890bcdf0b227513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:55 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:43776
I20260925 11:06:55.375072 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet 18efe3f13d054251890bcdf0b227513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:55 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:57214
I20260925 11:06:55.375135 13154 tablet_service.cc:1514] Processing DeleteTablet for tablet 18efe3f13d054251890bcdf0b227513f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:55 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:35840
I20260925 11:06:55.376046 13527 tablet_replica.cc:330] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: stopping tablet replica
I20260925 11:06:55.376585 13527 raft_consensus.cc:2229] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:55.377218 13527 raft_consensus.cc:2258] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:55.377408 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13092
I20260925 11:06:55.378389 13528 tablet_replica.cc:330] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: stopping tablet replica
I20260925 11:06:55.378938 13528 raft_consensus.cc:2229] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:55.379805 13528 raft_consensus.cc:2258] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:55.380054 13527 ts_tablet_manager.cc:1905] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:55.382018 13528 ts_tablet_manager.cc:1905] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:55.387293 13527 ts_tablet_manager.cc:1918] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:55.387501 13527 log.cc:1196] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/18efe3f13d054251890bcdf0b227513f
I20260925 11:06:55.388091 13527 ts_tablet_manager.cc:1939] T 18efe3f13d054251890bcdf0b227513f P daa5d5b674de43509944fcfed11dcb95: Deleting consensus metadata
I20260925 11:06:55.390206 13015 catalog_manager.cc:4841] TS daa5d5b674de43509944fcfed11dcb95 (127.8.181.2:41721): tablet 18efe3f13d054251890bcdf0b227513f (table test-table [id=caab2d6ca74e471a9d4e175c786a46e0]) successfully deleted
I20260925 11:06:55.391714 13528 ts_tablet_manager.cc:1918] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:55.391947 13529 tablet_replica.cc:330] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: stopping tablet replica
I20260925 11:06:55.391992 13528 log.cc:1196] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/18efe3f13d054251890bcdf0b227513f
I20260925 11:06:55.392468 13529 raft_consensus.cc:2229] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:55.392771 13528 ts_tablet_manager.cc:1939] T 18efe3f13d054251890bcdf0b227513f P ec3ba9fedcc04fcea84dcfed01af273e: Deleting consensus metadata
I20260925 11:06:55.393006 13529 raft_consensus.cc:2258] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:55.394791 13016 catalog_manager.cc:4841] TS ec3ba9fedcc04fcea84dcfed01af273e (127.8.181.3:45539): tablet 18efe3f13d054251890bcdf0b227513f (table test-table [id=caab2d6ca74e471a9d4e175c786a46e0]) successfully deleted
I20260925 11:06:55.396395 13529 ts_tablet_manager.cc:1905] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 11:06:55 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:06:55.404251 13529 ts_tablet_manager.cc:1918] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:55.404481 13529 log.cc:1196] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/18efe3f13d054251890bcdf0b227513f
I20260925 11:06:55.405138 13529 ts_tablet_manager.cc:1939] T 18efe3f13d054251890bcdf0b227513f P ed5e689480db43ee82889a67b91a3dca: Deleting consensus metadata
I20260925 11:06:55.407642 13018 catalog_manager.cc:4841] TS ed5e689480db43ee82889a67b91a3dca (127.8.181.1:34669): tablet 18efe3f13d054251890bcdf0b227513f (table test-table [id=caab2d6ca74e471a9d4e175c786a46e0]) successfully deleted
I20260925 11:06:55.408058 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.378913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.8.181.1:34669)
0925 11:06:55.379146 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.379152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.391270 (+ 12118us) server_negotiation.cc:407] Connection header received
0925 11:06:55.391473 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.391481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.391617 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.391760 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:55.393570 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.394819 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.396168 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.396681 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.396822 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.401082 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:55.401094 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:55.401110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:55.401154 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:55.404919 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.405804 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.405813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.405815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.405950 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.406359 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.406364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.406367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.406815 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:55.407155 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.407467 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.407711 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 11:06:55.409217 13214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:55.505255 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13092
I20260925 11:06:55.523663 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13225
Sep 25 11:06:55 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:06:55.544701 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.524622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.8.181.2:41721)
0925 11:06:55.524807 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.524814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.524836 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:06:55.525004 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.525010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.525118 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.525229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:55.526912 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.528102 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.529967 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.530736 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.530880 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.536631 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:55.536652 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:55.536674 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:55.536729 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:55.541255 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.542255 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.542266 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.542270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.542447 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.542865 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.542870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.542876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.543483 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:55.543766 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.544076 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.544326 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 11:06:55.545503 13347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:55.640187 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13225
I20260925 11:06:55.657912 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13358
Sep 25 11:06:55 dist-test-slave-6042 krb5kdc[12987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:06:55.676092 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:55.658888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.8.181.3:45539)
0925 11:06:55.659088 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:55.659096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:55.659118 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:06:55.659337 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:55.659343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:55.659415 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:55.659528 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:55.660860 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.661711 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.663234 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:55.663716 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:55.663914 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:55.668870 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:55.668887 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:55.668908 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:06:55.668959 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:55.672571 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.673521 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.673530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.673540 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.673727 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:06:55.674176 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:06:55.674181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:06:55.674186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:06:55.674743 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:55.675084 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:55.675421 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 11:06:55.675660 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:06:55.676834 13480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:55.803293 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13358
I20260925 11:06:55.823835 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13000
I20260925 11:06:55.825070 13061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:55.937366 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13000
2026-09-25T11:06:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4455 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T11:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:55Z Disabled control of system clock
I20260925 11:06:56.010118 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41775
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:41775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 11:06:56.358326 13538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:56.358768 13538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:56.358966 13538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:56.367187 13538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:56.367334 13538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:56.367400 13538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:56.367488 13538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:56.367611 13538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:06:56.377478 13538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:41775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:56.378935 13538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:56.380822 13538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:56.393680 13546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:56.393711 13544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:56.393926 13543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:56.394862 13538 server_base.cc:1034] running on GCE node
I20260925 11:06:56.395671 13538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:56.397156 13538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:56.398371 13538 hybrid_clock.cc:648] HybridClock initialized: now 1790334416398328 us; error 32 us; skew 500 ppm
I20260925 11:06:56.398772 13538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:06:56.407835 13538 webserver.cc:460] Webserver started at http://127.8.181.62:35403/ using document root <none> and password file <none>
I20260925 11:06:56.408633 13538 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:56.408735 13538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:56.409026 13538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:56.412230 13538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "c6674eb54c604c82977f4a981ebf70ae"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "b1abf52de9d2b7bcdbe7d2a5d255e5f6"
server_key_iv: "1ecf43d56ff3ab23be42951506bff63d"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.413005 13538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "c6674eb54c604c82977f4a981ebf70ae"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "b1abf52de9d2b7bcdbe7d2a5d255e5f6"
server_key_iv: "1ecf43d56ff3ab23be42951506bff63d"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.418601 13538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 11:06:56.422312 13552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:56.423981 13538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:56.424142 13538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "c6674eb54c604c82977f4a981ebf70ae"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "b1abf52de9d2b7bcdbe7d2a5d255e5f6"
server_key_iv: "1ecf43d56ff3ab23be42951506bff63d"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.424338 13538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:56.430346 13538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:56.431617 13538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:56.431923 13538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:56.451211 13538 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:41775
I20260925 11:06:56.451304 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:41775 every 8 connection(s)
I20260925 11:06:56.452607 13538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:56.453295 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13538
I20260925 11:06:56.453509 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:56.454427 8916 external_mini_cluster.cc:1428] Setting key 9b81df07c3f89d96f1cdf88ff87fcfdc
I20260925 11:06:56.457228 13604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:56.463541 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:56.456656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.8.181.62:41775)
0925 11:06:56.457806 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:56.457817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:56.457860 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:06:56.459651 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:56.459674 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:56.460116 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:56.460495 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:56.461699 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:56.461706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:56.461720 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:56.461736 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:56.461818 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:56.461835 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:56.462025 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:56.462138 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 11:06:56.462515 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":458,"threads_started":1}
I20260925 11:06:56.470063 13604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: Bootstrap starting.
I20260925 11:06:56.473253 13604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:56.474324 13604 log.cc:824] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:56.477582 13604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: No bootstrap required, opened a new log
I20260925 11:06:56.485788 13604 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } }
I20260925 11:06:56.486233 13604 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:56.486346 13604 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6674eb54c604c82977f4a981ebf70ae, State: Initialized, Role: FOLLOWER
I20260925 11:06:56.486984 13604 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } }
I20260925 11:06:56.487236 13604 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:56.487393 13604 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:56.487581 13604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:56.489836 13604 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } }
I20260925 11:06:56.490382 13604 leader_election.cc:302] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6674eb54c604c82977f4a981ebf70ae; no voters:
I20260925 11:06:56.491127 13604 leader_election.cc:288] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:56.491331 13609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:56.493307 13609 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [term 1 LEADER]: Becoming Leader. State: Replica: c6674eb54c604c82977f4a981ebf70ae, State: Running, Role: LEADER
I20260925 11:06:56.493805 13604 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:56.493912 13609 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } }
I20260925 11:06:56.497396 13611 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6674eb54c604c82977f4a981ebf70ae. Latest consensus state: current_term: 1 leader_uuid: "c6674eb54c604c82977f4a981ebf70ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } } }
I20260925 11:06:56.497473 13610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6674eb54c604c82977f4a981ebf70ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6674eb54c604c82977f4a981ebf70ae" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41775 } } }
I20260925 11:06:56.497831 13611 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:56.497862 13610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:56.498852 13619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:56.504773 13619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:56.514454 13619 catalog_manager.cc:1347] Generated new cluster ID: 2b72ea3200a7432997239bcbf3a5ed9a
I20260925 11:06:56.514675 13619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:56.529064 13619 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:56.529852 13619 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:56.543542 13619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: Generated new TSK 0
I20260925 11:06:56.544791 13619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:06:56.553007 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--builtin_ntp_servers=127.8.181.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 11:06:56.931862 13628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:56.932160 13628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:56.932255 13628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:56.939934 13628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:56.940049 13628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:56.940160 13628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:06:56.949630 13628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:56.950965 13628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:56.952718 13628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:56.967157 13633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:56.967281 13636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:56.967248 13634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:56.968696 13628 server_base.cc:1034] running on GCE node
I20260925 11:06:56.969367 13628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:56.970539 13628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:56.971798 13628 hybrid_clock.cc:648] HybridClock initialized: now 1790334416971755 us; error 33 us; skew 500 ppm
I20260925 11:06:56.972153 13628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:06:56.974731 13628 webserver.cc:460] Webserver started at http://127.8.181.1:45917/ using document root <none> and password file <none>
I20260925 11:06:56.975495 13628 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:56.975646 13628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:56.976039 13628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:56.979228 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "066c2b6320e04f61bf84439b4ce7510e"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "bfab41ddc8d9ab8d3aa279e975894026"
server_key_iv: "d664da21c99a5a01111a7a6d368e47f3"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.980079 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "066c2b6320e04f61bf84439b4ce7510e"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "bfab41ddc8d9ab8d3aa279e975894026"
server_key_iv: "d664da21c99a5a01111a7a6d368e47f3"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.985728 13628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 11:06:56.989794 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:56.991472 13628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:06:56.991676 13628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "066c2b6320e04f61bf84439b4ce7510e"
format_stamp: "Formatted at 2026-09-25 11:06:56 on dist-test-slave-6042"
server_key: "bfab41ddc8d9ab8d3aa279e975894026"
server_key_iv: "d664da21c99a5a01111a7a6d368e47f3"
server_key_version: "encryptionkey@0"
I20260925 11:06:56.991940 13628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:56.998167 13628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:56.999119 13628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:56.999408 13628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:57.000377 13628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:57.001889 13628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:57.002014 13628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.002259 13628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:57.002408 13628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.039609 13628 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:39703
I20260925 11:06:57.039660 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:39703 every 8 connection(s)
I20260925 11:06:57.041060 13628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:06:57.047886 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13628
I20260925 11:06:57.048092 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:06:57.048892 8916 external_mini_cluster.cc:1428] Setting key 95816bf7e2f381a7108853c35fa36a0c
I20260925 11:06:57.051257 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.044097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:55019 (local address 127.8.181.62:41775)
0925 11:06:57.044569 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:57.044577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:57.045842 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 11:06:57.047712 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:57.047716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:57.047775 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:57.047869 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:57.049607 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:57.049613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:57.049620 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:57.049642 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:57.049762 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:57.049788 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:57.049929 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:57.050824 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 11:06:57.050905 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260925 11:06:57.051664 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.044744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41775 (local address 127.8.181.1:55019)
0925 11:06:57.045637 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:57.045715 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:57.047356 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:57.048160 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:57.048174 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:57.048304 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:57.049207 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:57.049211 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:57.049259 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:57.049385 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:57.049907 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:57.049917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:57.049925 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 11:06:57.050226 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 11:06:57.050432 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":274,"threads_started":1}
I20260925 11:06:57.052212 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--builtin_ntp_servers=127.8.181.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 11:06:57.053213 13755 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41775
I20260925 11:06:57.053527 13755 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:57.054209 13755 heartbeater.cc:511] Master 127.8.181.62:41775 requested a full tablet report, sending...
I20260925 11:06:57.056494 13569 ts_manager.cc:194] Registered new tserver with Master: 066c2b6320e04f61bf84439b4ce7510e (127.8.181.1:39703)
W20260925 11:06:57.384155 13759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:57.384435 13759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:57.384582 13759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:57.392266 13759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:57.392371 13759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:57.392537 13759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:06:57.402357 13759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:57.403707 13759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:57.405504 13759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:57.419531 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:57.419615 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:57.419597 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:57.424237 13759 server_base.cc:1034] running on GCE node
I20260925 11:06:57.424953 13759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:57.425947 13759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:57.427188 13759 hybrid_clock.cc:648] HybridClock initialized: now 1790334417427120 us; error 53 us; skew 500 ppm
I20260925 11:06:57.427515 13759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:06:57.429955 13759 webserver.cc:460] Webserver started at http://127.8.181.2:43495/ using document root <none> and password file <none>
I20260925 11:06:57.430778 13759 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:57.430907 13759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:57.431226 13759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:57.434309 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "100a50c5da4a5146c6d7c231498ccc9d"
server_key_iv: "6e131fb18cac604d2c7e894c3293ee65"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.435176 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "100a50c5da4a5146c6d7c231498ccc9d"
server_key_iv: "6e131fb18cac604d2c7e894c3293ee65"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.440650 13759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:57.444435 13773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.446106 13759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:06:57.446308 13759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "100a50c5da4a5146c6d7c231498ccc9d"
server_key_iv: "6e131fb18cac604d2c7e894c3293ee65"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.446628 13759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:57.452702 13759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:57.453516 13759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:57.453862 13759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:57.454995 13759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:57.456503 13759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:57.456634 13759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.456771 13759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:57.456871 13759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.492005 13759 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:42217
I20260925 11:06:57.492039 13885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:42217 every 8 connection(s)
I20260925 11:06:57.493299 13759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:06:57.494935 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13759
I20260925 11:06:57.495102 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:06:57.495603 8916 external_mini_cluster.cc:1428] Setting key 3a207aeff0607b6cecfde81b63a6e6b7
I20260925 11:06:57.498900 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--builtin_ntp_servers=127.8.181.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 11:06:57.503010 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.496350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35581 (local address 127.8.181.62:41775)
0925 11:06:57.496699 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:57.496705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:57.498326 (+ 1621us) server_negotiation.cc:407] Connection header received
0925 11:06:57.500240 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:57.500250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:57.500325 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:57.500446 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:57.502061 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:57.502070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:57.502077 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:57.502102 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:57.502146 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:57.502164 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:57.502324 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:57.502598 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 11:06:57.502672 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260925 11:06:57.504024 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.496956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41775 (local address 127.8.181.2:35581)
0925 11:06:57.498141 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:57.498197 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:57.499921 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:57.500699 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:57.500709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:57.500811 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:57.501730 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:57.501733 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:57.501773 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:57.501863 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:57.502303 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:57.502313 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:57.502320 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 11:06:57.502664 (+ 344us) client_negotiation.cc:240] Negotiation successful
0925 11:06:57.502893 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":409,"threads_started":1}
I20260925 11:06:57.505721 13886 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41775
I20260925 11:06:57.506098 13886 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:57.506839 13886 heartbeater.cc:511] Master 127.8.181.62:41775 requested a full tablet report, sending...
I20260925 11:06:57.508491 13569 ts_manager.cc:194] Registered new tserver with Master: bbbf00c9a3d94a8e9d1cc77cff17ce8a (127.8.181.2:42217)
W20260925 11:06:57.840643 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:57.841030 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:57.841236 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:57.848673 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:57.848783 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:57.848968 13889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:06:57.858359 13889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:57.859711 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:57.861644 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:57.875197 13895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:57.875351 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:57.875478 13894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:57.876770 13889 server_base.cc:1034] running on GCE node
I20260925 11:06:57.877450 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:57.878747 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:57.880039 13889 hybrid_clock.cc:648] HybridClock initialized: now 1790334417879961 us; error 75 us; skew 500 ppm
I20260925 11:06:57.880401 13889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:06:57.883162 13889 webserver.cc:460] Webserver started at http://127.8.181.3:35685/ using document root <none> and password file <none>
I20260925 11:06:57.883924 13889 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:57.884066 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:57.884408 13889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:57.887495 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "beceb2e1ac424ff38c90b93babe985cb"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "190886e094da90f51c031ebbfd6d94d3"
server_key_iv: "13fa64155b66bd711b759149131f12e7"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.888379 13889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "beceb2e1ac424ff38c90b93babe985cb"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "190886e094da90f51c031ebbfd6d94d3"
server_key_iv: "13fa64155b66bd711b759149131f12e7"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.894730 13889 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 11:06:57.898931 13903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.900477 13889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:06:57.900852 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "beceb2e1ac424ff38c90b93babe985cb"
format_stamp: "Formatted at 2026-09-25 11:06:57 on dist-test-slave-6042"
server_key: "190886e094da90f51c031ebbfd6d94d3"
server_key_iv: "13fa64155b66bd711b759149131f12e7"
server_key_version: "encryptionkey@0"
I20260925 11:06:57.901124 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:57.907143 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:57.908037 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:57.908320 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:57.909297 13889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:06:57.910972 13889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:06:57.911108 13889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.911247 13889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:06:57.911331 13889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:57.947129 13889 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:36573
I20260925 11:06:57.947222 14015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:36573 every 8 connection(s)
I20260925 11:06:57.948532 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:06:57.953608 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 13889
I20260925 11:06:57.953785 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:06:57.954329 8916 external_mini_cluster.cc:1428] Setting key 3322accabef0badf36293491d747bef9
I20260925 11:06:57.958669 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.951391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:53317 (local address 127.8.181.62:41775)
0925 11:06:57.951632 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:57.951640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:57.953202 (+ 1562us) server_negotiation.cc:407] Connection header received
0925 11:06:57.955707 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:57.955713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:57.955795 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:57.955905 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:57.957808 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:57.957814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:57.957818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:57.957838 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:57.957878 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:57.957895 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:57.958032 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:57.958289 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 11:06:57.958349 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:06:57.959821 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.951979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41775 (local address 127.8.181.3:53317)
0925 11:06:57.952990 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:57.953033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:57.955275 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:57.956227 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:57.956241 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:57.956381 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:57.957357 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:57.957360 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:57.957445 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:57.957565 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:57.958036 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:57.958048 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:57.958058 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 11:06:57.958354 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 11:06:57.958676 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":286,"threads_started":1}
I20260925 11:06:57.961418 14016 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41775
I20260925 11:06:57.961824 14016 heartbeater.cc:464] Registering TS with master...
I20260925 11:06:57.962414 14016 heartbeater.cc:511] Master 127.8.181.62:41775 requested a full tablet report, sending...
I20260925 11:06:57.963974 13569 ts_manager.cc:194] Registered new tserver with Master: beceb2e1ac424ff38c90b93babe985cb (127.8.181.3:36573)
I20260925 11:06:57.971333 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:06:57.979262 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:57.976369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.8.181.62:41775)
0925 11:06:57.976655 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:57.976660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:57.977063 (+ 403us) server_negotiation.cc:407] Connection header received
0925 11:06:57.977244 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:57.977249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:57.977309 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:57.977414 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:57.978585 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:57.978590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:57.978593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:57.978611 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:57.978704 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:57.978719 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:57.978845 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:57.978980 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 11:06:57.979038 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 11:06:57.984543 13569 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:06:57.992735 13569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:06:58.024025 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.007599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:39703 (local address 127.0.0.1:50784)
0925 11:06:58.018021 (+ 10422us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:58.018104 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:58.018342 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:58.020191 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:58.020204 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:58.020359 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:58.022267 (+ 1908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:58.022272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:58.022310 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:58.022425 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:58.023176 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:58.023191 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:58.023199 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 11:06:58.023427 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 11:06:58.023529 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9974,"thread_start_us":15156,"threads_started":1}
I20260925 11:06:58.024019 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.007803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.8.181.1:39703)
0925 11:06:58.019195 (+ 11392us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.019206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.019242 (+ 36us) server_negotiation.cc:407] Connection header received
0925 11:06:58.019320 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.019328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.019590 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.020189 (+ 599us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.022896 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.022907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.022926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.022958 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.023025 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.023049 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.023193 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.023418 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.023535 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11252,"thread_start_us":11104,"threads_started":1}
I20260925 11:06:58.026924 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.011176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.8.181.3:36573)
0925 11:06:58.022564 (+ 11388us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.022574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.023203 (+ 629us) server_negotiation.cc:407] Connection header received
0925 11:06:58.023463 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.023471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.023694 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.023828 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.025934 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.025944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.025961 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.025986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.026043 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.026059 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.026286 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.026366 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.026498 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11119,"thread_start_us":10900,"threads_started":1}
I20260925 11:06:58.026950 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.010578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:36573 (local address 127.0.0.1:51454)
0925 11:06:58.023055 (+ 12477us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:58.023075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:58.023288 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:58.024770 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:58.024775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:58.024803 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:58.025668 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:58.025675 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:58.025696 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:58.025754 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:58.026183 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:58.026189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:58.026194 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 11:06:58.026333 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 11:06:58.026397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12325,"thread_start_us":6780,"threads_started":1}
I20260925 11:06:58.028169 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.008123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.8.181.2:42217)
0925 11:06:58.018576 (+ 10453us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.018587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.022331 (+ 3744us) server_negotiation.cc:407] Connection header received
0925 11:06:58.022411 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.022417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.022705 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.023983 (+ 1278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.027191 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.027205 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.027221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.027243 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.027303 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.027326 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.027453 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.027717 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.027825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10244,"thread_start_us":10455,"threads_started":1}
I20260925 11:06:58.028223 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.007896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:42217 (local address 127.0.0.1:44494)
0925 11:06:58.022016 (+ 14120us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:58.022057 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:58.022194 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:58.025001 (+ 2807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:58.025006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:58.025050 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:58.026664 (+ 1614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:58.026667 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:58.026685 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:58.026741 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:58.027507 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:58.027512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:58.027517 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 11:06:58.027670 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 11:06:58.027737 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13927,"thread_start_us":13722,"threads_started":1}
I20260925 11:06:58.028817 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 (DEFAULT_TABLE table=test-table [id=2ea272414603472ea49f1e97581ebe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:58.030364 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9b858f98aa46ebad28d2e3f103a325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:58.031765 13951 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 (DEFAULT_TABLE table=test-table [id=2ea272414603472ea49f1e97581ebe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:58.032510 13821 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 (DEFAULT_TABLE table=test-table [id=2ea272414603472ea49f1e97581ebe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:06:58.033288 13951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9b858f98aa46ebad28d2e3f103a325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:58.033892 13821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9b858f98aa46ebad28d2e3f103a325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:58.052383 14034 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Bootstrap starting.
I20260925 11:06:58.055207 14036 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Bootstrap starting.
I20260925 11:06:58.056022 14035 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Bootstrap starting.
I20260925 11:06:58.056453 14034 tablet_bootstrap.cc:654] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:58.057868 14034 log.cc:824] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:58.059793 13755 heartbeater.cc:503] Master 127.8.181.62:41775 was elected leader, sending a full tablet report...
I20260925 11:06:58.060109 14036 tablet_bootstrap.cc:654] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:58.060747 14035 tablet_bootstrap.cc:654] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:58.061352 14036 log.cc:824] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:58.061911 14034 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: No bootstrap required, opened a new log
I20260925 11:06:58.061998 14035 log.cc:824] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:58.062337 14034 ts_tablet_manager.cc:1397] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260925 11:06:58.071305 14036 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: No bootstrap required, opened a new log
I20260925 11:06:58.071638 14036 ts_tablet_manager.cc:1397] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260925 11:06:58.072365 14035 tablet_bootstrap.cc:492] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: No bootstrap required, opened a new log
I20260925 11:06:58.072672 14035 ts_tablet_manager.cc:1397] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260925 11:06:58.074726 14034 raft_consensus.cc:348] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.075242 14034 raft_consensus.cc:374] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:58.075392 14034 raft_consensus.cc:729] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 066c2b6320e04f61bf84439b4ce7510e, State: Initialized, Role: FOLLOWER
I20260925 11:06:58.076120 14034 consensus_queue.cc:260] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.078094 14034 ts_tablet_manager.cc:1428] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260925 11:06:58.082252 14036 raft_consensus.cc:348] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.082861 14036 raft_consensus.cc:374] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:58.082979 14036 raft_consensus.cc:729] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beceb2e1ac424ff38c90b93babe985cb, State: Initialized, Role: FOLLOWER
I20260925 11:06:58.083154 14035 raft_consensus.cc:348] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.083662 14035 raft_consensus.cc:374] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:58.083797 14035 raft_consensus.cc:729] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbf00c9a3d94a8e9d1cc77cff17ce8a, State: Initialized, Role: FOLLOWER
I20260925 11:06:58.083675 14036 consensus_queue.cc:260] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.084394 14035 consensus_queue.cc:260] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.084890 14016 heartbeater.cc:503] Master 127.8.181.62:41775 was elected leader, sending a full tablet report...
I20260925 11:06:58.085570 13886 heartbeater.cc:503] Master 127.8.181.62:41775 was elected leader, sending a full tablet report...
I20260925 11:06:58.086141 14035 ts_tablet_manager.cc:1428] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 11:06:58.086792 14036 ts_tablet_manager.cc:1428] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260925 11:06:58.200299 14017 tablet.cc:2369] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:06:58.246650 13887 tablet.cc:2369] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:58.290880 14041 raft_consensus.cc:482] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:06:58.291196 14041 raft_consensus.cc:504] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.293042 14041 leader_election.cc:288] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 066c2b6320e04f61bf84439b4ce7510e (127.8.181.1:39703), bbbf00c9a3d94a8e9d1cc77cff17ce8a (127.8.181.2:42217)
W20260925 11:06:58.294071 13756 tablet.cc:2369] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:06:58.296723 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.293688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:43197 (local address 127.8.181.2:42217)
0925 11:06:58.294031 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.294040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.294381 (+ 341us) server_negotiation.cc:407] Connection header received
0925 11:06:58.294458 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.294494 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.294566 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.294674 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.296053 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.296060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.296064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.296090 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.296145 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.296162 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.296307 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.296383 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.296460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260925 11:06:58.296739 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.293629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:42217 (local address 127.8.181.3:43197)
0925 11:06:58.294013 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:58.294038 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:58.294234 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:58.294948 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:58.294953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:58.294980 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:58.295811 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:58.295815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:58.295839 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:58.295894 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:58.296275 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:58.296280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:58.296284 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 11:06:58.296390 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:06:58.296443 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260925 11:06:58.297020 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.293584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:60433 (local address 127.8.181.1:39703)
0925 11:06:58.293822 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.293829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.294787 (+ 958us) server_negotiation.cc:407] Connection header received
0925 11:06:58.294961 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.294967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.295033 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.295174 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.296279 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.296284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.296291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.296314 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.296358 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.296371 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.296508 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.296646 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.296714 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:06:58.297286 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.293687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:39703 (local address 127.8.181.3:60433)
0925 11:06:58.294633 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:58.294652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:58.294808 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:58.295294 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:58.295300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:58.295329 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:06:58.295998 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:06:58.296001 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:06:58.296014 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:06:58.296052 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:06:58.296478 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:06:58.296482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:06:58.296487 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 11:06:58.296720 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 11:06:58.296777 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":576,"threads_started":1}
I20260925 11:06:58.297851 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9b858f98aa46ebad28d2e3f103a325" candidate_uuid: "beceb2e1ac424ff38c90b93babe985cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" is_pre_election: true
I20260925 11:06:58.298542 13841 raft_consensus.cc:2454] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beceb2e1ac424ff38c90b93babe985cb in term 0.
I20260925 11:06:58.298408 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9b858f98aa46ebad28d2e3f103a325" candidate_uuid: "beceb2e1ac424ff38c90b93babe985cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066c2b6320e04f61bf84439b4ce7510e" is_pre_election: true
I20260925 11:06:58.298983 13710 raft_consensus.cc:2454] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beceb2e1ac424ff38c90b93babe985cb in term 0.
I20260925 11:06:58.299367 13907 leader_election.cc:302] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbbf00c9a3d94a8e9d1cc77cff17ce8a, beceb2e1ac424ff38c90b93babe985cb; no voters:
I20260925 11:06:58.299928 14041 raft_consensus.cc:2781] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:06:58.300091 14041 raft_consensus.cc:482] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:06:58.300194 14041 raft_consensus.cc:3037] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:58.302368 14041 raft_consensus.cc:504] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.303153 14041 leader_election.cc:288] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [CANDIDATE]: Term 1 election: Requested vote from peers 066c2b6320e04f61bf84439b4ce7510e (127.8.181.1:39703), bbbf00c9a3d94a8e9d1cc77cff17ce8a (127.8.181.2:42217)
I20260925 11:06:58.303797 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9b858f98aa46ebad28d2e3f103a325" candidate_uuid: "beceb2e1ac424ff38c90b93babe985cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a"
I20260925 11:06:58.303821 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9b858f98aa46ebad28d2e3f103a325" candidate_uuid: "beceb2e1ac424ff38c90b93babe985cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "066c2b6320e04f61bf84439b4ce7510e"
I20260925 11:06:58.304050 13710 raft_consensus.cc:3037] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:58.304050 13841 raft_consensus.cc:3037] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:58.307524 13841 raft_consensus.cc:2454] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beceb2e1ac424ff38c90b93babe985cb in term 1.
I20260925 11:06:58.307523 13710 raft_consensus.cc:2454] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beceb2e1ac424ff38c90b93babe985cb in term 1.
I20260925 11:06:58.308099 13904 leader_election.cc:302] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 066c2b6320e04f61bf84439b4ce7510e, beceb2e1ac424ff38c90b93babe985cb; no voters:
I20260925 11:06:58.308729 14041 raft_consensus.cc:2781] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:58.309278 14041 raft_consensus.cc:686] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 1 LEADER]: Becoming Leader. State: Replica: beceb2e1ac424ff38c90b93babe985cb, State: Running, Role: LEADER
I20260925 11:06:58.309887 14041 consensus_queue.cc:237] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } }
I20260925 11:06:58.315021 13569 catalog_manager.cc:5495] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb reported cstate change: term changed from 0 to 1, leader changed from <none> to beceb2e1ac424ff38c90b93babe985cb (127.8.181.3). New cstate: current_term: 1 leader_uuid: "beceb2e1ac424ff38c90b93babe985cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "beceb2e1ac424ff38c90b93babe985cb" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36573 } health_report { overall_health: HEALTHY } } }
I20260925 11:06:58.335289 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.332904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.8.181.3:36573)
0925 11:06:58.333080 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.333085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.333201 (+ 116us) server_negotiation.cc:407] Connection header received
0925 11:06:58.333386 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.333389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.333476 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.333568 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.334666 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.334670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.334674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.334689 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.334751 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.334765 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.334873 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.335046 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.335096 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:06:58.344213 13841 raft_consensus.cc:1261] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 1 FOLLOWER]: Refusing update from remote peer beceb2e1ac424ff38c90b93babe985cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:58.344213 13710 raft_consensus.cc:1261] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 1 FOLLOWER]: Refusing update from remote peer beceb2e1ac424ff38c90b93babe985cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:06:58.345461 14041 consensus_queue.cc:1035] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbf00c9a3d94a8e9d1cc77cff17ce8a" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:58.345811 14044 consensus_queue.cc:1035] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "066c2b6320e04f61bf84439b4ce7510e" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:06:58.361158 14047 mvcc.cc:205] Tried to move back new op lower bound from 7333209777527365632 to 7333209777401978880. Current Snapshot: MvccSnapshot[applied={T|T < 7333209777527365632}]
I20260925 11:06:58.363903 14049 mvcc.cc:205] Tried to move back new op lower bound from 7333209777527365632 to 7333209777401978880. Current Snapshot: MvccSnapshot[applied={T|T < 7333209777527365632}]
I20260925 11:06:58.383184 13568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:58.383523 13568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:06:58.387753 13568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6674eb54c604c82977f4a981ebf70ae: Sending DeleteTablet for 3 replicas of tablet 7e9b858f98aa46ebad28d2e3f103a325
I20260925 11:06:58.389283 13821 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:58 UTC) from {username='slave'} at 127.0.0.1:44494
I20260925 11:06:58.389285 13690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:58 UTC) from {username='slave'} at 127.0.0.1:50784
I20260925 11:06:58.390113 13951 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e9b858f98aa46ebad28d2e3f103a325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:06:58 UTC) from {username='slave'} at 127.0.0.1:51454
I20260925 11:06:58.390523 14059 tablet_replica.cc:330] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: stopping tablet replica
I20260925 11:06:58.391004 14059 raft_consensus.cc:2229] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:58.392127 14059 raft_consensus.cc:2258] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:58.392705 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13628
I20260925 11:06:58.393093 14060 tablet_replica.cc:330] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: stopping tablet replica
I20260925 11:06:58.393574 14060 raft_consensus.cc:2229] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:06:58.394546 14060 raft_consensus.cc:2258] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:58.395888 14059 ts_tablet_manager.cc:1905] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:58.402364 14060 ts_tablet_manager.cc:1905] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:58.403822 14059 ts_tablet_manager.cc:1918] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:58.404126 14059 log.cc:1196] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/7e9b858f98aa46ebad28d2e3f103a325
I20260925 11:06:58.404785 14059 ts_tablet_manager.cc:1939] T 7e9b858f98aa46ebad28d2e3f103a325 P 066c2b6320e04f61bf84439b4ce7510e: Deleting consensus metadata
I20260925 11:06:58.406585 14058 tablet_replica.cc:330] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: stopping tablet replica
I20260925 11:06:58.407266 14058 raft_consensus.cc:2229] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:06:58.407328 13553 catalog_manager.cc:4841] TS 066c2b6320e04f61bf84439b4ce7510e (127.8.181.1:39703): tablet 7e9b858f98aa46ebad28d2e3f103a325 (table test-table [id=2ea272414603472ea49f1e97581ebe37]) successfully deleted
I20260925 11:06:58.407900 14058 raft_consensus.cc:2258] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:06:58.410521 14058 ts_tablet_manager.cc:1905] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:06:58.413723 14060 ts_tablet_manager.cc:1918] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:58.414026 14060 log.cc:1196] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/7e9b858f98aa46ebad28d2e3f103a325
I20260925 11:06:58.414755 14060 ts_tablet_manager.cc:1939] T 7e9b858f98aa46ebad28d2e3f103a325 P beceb2e1ac424ff38c90b93babe985cb: Deleting consensus metadata
I20260925 11:06:58.415088 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.394173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.8.181.1:39703)
0925 11:06:58.394382 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.394388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.412197 (+ 17809us) server_negotiation.cc:407] Connection header received
0925 11:06:58.412728 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.412735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.412846 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.412925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.414232 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.414240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.414246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.414268 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.414306 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.414322 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.414427 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.414719 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.414792 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:06:58.416368 13750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:58.416992 13555 catalog_manager.cc:4841] TS beceb2e1ac424ff38c90b93babe985cb (127.8.181.3:36573): tablet 7e9b858f98aa46ebad28d2e3f103a325 (table test-table [id=2ea272414603472ea49f1e97581ebe37]) successfully deleted
I20260925 11:06:58.419008 14058 ts_tablet_manager.cc:1918] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:06:58.419412 14058 log.cc:1196] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/7e9b858f98aa46ebad28d2e3f103a325
I20260925 11:06:58.420357 14058 ts_tablet_manager.cc:1939] T 7e9b858f98aa46ebad28d2e3f103a325 P bbbf00c9a3d94a8e9d1cc77cff17ce8a: Deleting consensus metadata
I20260925 11:06:58.422636 13556 catalog_manager.cc:4841] TS bbbf00c9a3d94a8e9d1cc77cff17ce8a (127.8.181.2:42217): tablet 7e9b858f98aa46ebad28d2e3f103a325 (table test-table [id=2ea272414603472ea49f1e97581ebe37]) successfully deleted
I20260925 11:06:58.538740 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13628
I20260925 11:06:58.557116 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13759
I20260925 11:06:58.560562 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.558031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.8.181.2:42217)
0925 11:06:58.558208 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.558215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.558268 (+ 53us) server_negotiation.cc:407] Connection header received
0925 11:06:58.558548 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.558553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.558644 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.558760 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.559856 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.559862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.559866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.559887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.559924 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.559938 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.560143 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.560197 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.560258 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:06:58.561408 13881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:58.654673 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13759
I20260925 11:06:58.671880 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13889
I20260925 11:06:58.675415 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:58.672716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.8.181.3:36573)
0925 11:06:58.672954 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:58.672961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:58.672985 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:06:58.673168 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:58.673175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:58.673248 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:58.673386 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:58.674545 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:58.674551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:58.674556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:58.674579 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:58.674622 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:58.674638 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:58.674805 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:58.674993 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 11:06:58.675069 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 11:06:58.676169 14011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:58.804097 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13889
I20260925 11:06:58.823875 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 13538
I20260925 11:06:58.825202 13599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:58.936283 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 13538
2026-09-25T11:06:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2996 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 11:06:58.977291 8916 mini_oidc.cc:152] Starting JWKS server
I20260925 11:06:58.979897 8916 webserver.cc:460] Webserver started at https://127.0.0.1:40863/ using document root <none> and password file <none>
I20260925 11:06:58.980263 8916 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 11:06:58.980957 8916 webserver.cc:460] Webserver started at http://127.0.0.1:34153/ using document root <none> and password file <none>
2026-09-25T11:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:58Z Disabled control of system clock
I20260925 11:06:59.011044 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38491
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:38491
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40863/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/testchainca.cert with env {}
W20260925 11:06:59.340005 14071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:06:59.340382 14071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:06:59.340552 14071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:06:59.348150 14071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:06:59.348317 14071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:06:59.348410 14071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:06:59.348522 14071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:06:59.348609 14071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 11:06:59.348701 14071 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 11:06:59.348780 14071 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40863/jwks.json
I20260925 11:06:59.359143 14071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:38491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40863/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:06:59.360612 14071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:06:59.361889 14071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:06:59.374401 14077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:59.374550 14076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:06:59.374537 14079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:06:59.375368 14071 server_base.cc:1034] running on GCE node
I20260925 11:06:59.376183 14071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:06:59.377758 14071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:06:59.378983 14071 hybrid_clock.cc:648] HybridClock initialized: now 1790334419378935 us; error 45 us; skew 500 ppm
I20260925 11:06:59.379392 14071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:06:59.381891 14071 webserver.cc:460] Webserver started at http://127.8.181.62:39733/ using document root <none> and password file <none>
I20260925 11:06:59.382702 14071 fs_manager.cc:359] Metadata directory not provided
I20260925 11:06:59.382839 14071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:06:59.383159 14071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:06:59.386152 14071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "771c12015c144d4ab26fd7e110b26c3a"
format_stamp: "Formatted at 2026-09-25 11:06:59 on dist-test-slave-6042"
server_key: "9765523a7a11a47dba4dbe8141f9aeef"
server_key_iv: "697fa34d28fd2c8c6d945e8035f582e2"
server_key_version: "encryptionkey@0"
I20260925 11:06:59.387001 14071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "771c12015c144d4ab26fd7e110b26c3a"
format_stamp: "Formatted at 2026-09-25 11:06:59 on dist-test-slave-6042"
server_key: "9765523a7a11a47dba4dbe8141f9aeef"
server_key_iv: "697fa34d28fd2c8c6d945e8035f582e2"
server_key_version: "encryptionkey@0"
I20260925 11:06:59.392417 14071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:06:59.396275 14085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:06:59.397770 14071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:06:59.397984 14071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "771c12015c144d4ab26fd7e110b26c3a"
format_stamp: "Formatted at 2026-09-25 11:06:59 on dist-test-slave-6042"
server_key: "9765523a7a11a47dba4dbe8141f9aeef"
server_key_iv: "697fa34d28fd2c8c6d945e8035f582e2"
server_key_version: "encryptionkey@0"
I20260925 11:06:59.398231 14071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:06:59.425105 14071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:06:59.426011 14071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:06:59.426340 14071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:06:59.445791 14071 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:38491
I20260925 11:06:59.445843 14136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:38491 every 8 connection(s)
I20260925 11:06:59.447333 14071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:06:59.451586 14137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:06:59.451668 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14071
I20260925 11:06:59.451882 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:06:59.452678 8916 external_mini_cluster.cc:1428] Setting key bd4f7810503b8e57906794ab6bd384c5
I20260925 11:06:59.466256 14137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a: Bootstrap starting.
I20260925 11:06:59.468519 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:59.454822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.8.181.62:38491)
0925 11:06:59.455877 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:59.455891 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:59.455935 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:06:59.458215 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:59.458244 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:59.458922 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:59.459303 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:06:59.460095 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.461449 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.463504 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.464269 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.464416 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:59.465922 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:06:59.465931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:06:59.465946 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:06:59.466687 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:06:59.466777 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:06:59.466799 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:06:59.466902 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 11:06:59.467145 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 11:06:59.467523 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":395,"threads_started":1}
I20260925 11:06:59.469918 14137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a: Neither blocks nor log segments found. Creating new log.
I20260925 11:06:59.471115 14137 log.cc:824] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:06:59.472551 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:06:59.474036 14137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a: No bootstrap required, opened a new log
I20260925 11:06:59.481303 14137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } }
I20260925 11:06:59.481684 14137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:06:59.481781 14137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771c12015c144d4ab26fd7e110b26c3a, State: Initialized, Role: FOLLOWER
I20260925 11:06:59.482395 14137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } }
I20260925 11:06:59.482684 14137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:06:59.482825 14137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:06:59.482993 14137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:06:59.485136 14137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } }
I20260925 11:06:59.485725 14137 leader_election.cc:302] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771c12015c144d4ab26fd7e110b26c3a; no voters:
I20260925 11:06:59.486320 14137 leader_election.cc:288] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:06:59.486502 14142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:06:59.487212 14142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [term 1 LEADER]: Becoming Leader. State: Replica: 771c12015c144d4ab26fd7e110b26c3a, State: Running, Role: LEADER
I20260925 11:06:59.487766 14142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } }
I20260925 11:06:59.488564 14137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:06:59.490661 14144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771c12015c144d4ab26fd7e110b26c3a. Latest consensus state: current_term: 1 leader_uuid: "771c12015c144d4ab26fd7e110b26c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } } }
I20260925 11:06:59.490896 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771c12015c144d4ab26fd7e110b26c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771c12015c144d4ab26fd7e110b26c3a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38491 } } }
I20260925 11:06:59.491175 14144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:59.491214 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a [sys.catalog]: This master's current role is: LEADER
I20260925 11:06:59.493608 14150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:06:59.497978 14150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:06:59.507704 14150 catalog_manager.cc:1347] Generated new cluster ID: 0ab73285dbb54367bbf5a1a1d98f1b8d
I20260925 11:06:59.507843 14150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:06:59.527827 14150 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:06:59.529848 14150 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:06:59.537669 14150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771c12015c144d4ab26fd7e110b26c3a: Generated new TSK 0
I20260925 11:06:59.538755 14150 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 11:06:59.582406 14167 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 11:06:59.582644 14167 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 11:06:59.583626 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:06:59.561339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.8.181.62:38491)
0925 11:06:59.561502 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 11:06:59.561506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:06:59.562029 (+ 523us) server_negotiation.cc:407] Connection header received
0925 11:06:59.562371 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:06:59.562375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:06:59.562444 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:06:59.562678 (+ 234us) server_negotiation.cc:226] Negotiated authn=JWT
0925 11:06:59.563285 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.564119 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.566205 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.566776 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.566876 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:59.567538 (+ 662us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 11:06:59.583095 (+ 15557us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 11:06:59.583394 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 11:06:59.583777 14140 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 11:06:59.585770 14166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:06:59.561136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38491 (local address 127.0.0.1:43902)
0925 11:06:59.561911 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 11:06:59.561932 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:06:59.562156 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:06:59.562721 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:06:59.562727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:06:59.562739 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 11:06:59.563100 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:59.563112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.564265 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.564271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:59.566048 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:06:59.566064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:06:59.566957 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:06:59.566964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:06:59.567165 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:06:59.567256 (+ 91us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 11:06:59.583435 (+ 16179us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 11:06:59.583840 (+ 405us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.181.62:38491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":352,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.181.62:38491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40863/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 11:06:59.587656 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14071
I20260925 11:06:59.589004 14132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:06:59.685544 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14071
2026-09-25T11:06:59Z chronyd exiting
I20260925 11:06:59.736196 8916 test_util.cc:175] -----------------------------------------------
I20260925 11:06:59.736342 8916 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790334376327139-8916-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (763 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 11:06:59.740322 8916 mini_oidc.cc:152] Starting JWKS server
I20260925 11:06:59.742615 8916 webserver.cc:460] Webserver started at https://127.0.0.1:46651/ using document root <none> and password file <none>
I20260925 11:06:59.742835 8916 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 11:06:59.743487 8916 webserver.cc:460] Webserver started at http://127.0.0.1:42817/ using document root <none> and password file <none>
2026-09-25T11:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:06:59Z Disabled control of system clock
I20260925 11:06:59.771013 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34539
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/testchainca.cert with env {}
W20260925 11:07:00.099941 14174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:00.100296 14174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:00.100456 14174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:00.107980 14174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:00.108139 14174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:00.108225 14174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:00.108325 14174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:00.108405 14174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 11:07:00.108494 14174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 11:07:00.108572 14174 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46651/jwks.json
I20260925 11:07:00.118831 14174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:00.120265 14174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:00.121611 14174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:00.134338 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:00.134443 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:00.134416 14182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:00.135494 14174 server_base.cc:1034] running on GCE node
I20260925 11:07:00.136390 14174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:00.137904 14174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:00.139156 14174 hybrid_clock.cc:648] HybridClock initialized: now 1790334420139105 us; error 60 us; skew 500 ppm
I20260925 11:07:00.139575 14174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:00.142125 14174 webserver.cc:460] Webserver started at http://127.8.181.62:45229/ using document root <none> and password file <none>
I20260925 11:07:00.143055 14174 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:00.143184 14174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:00.143508 14174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:00.146956 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "d50de408c88141dc8e6d060b7aa0558a"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "9efab42c253d44c7e1482838be587960"
server_key_iv: "d283115fc6d98f410741aeef0b75b5b7"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.147776 14174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "d50de408c88141dc8e6d060b7aa0558a"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "9efab42c253d44c7e1482838be587960"
server_key_iv: "d283115fc6d98f410741aeef0b75b5b7"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.153190 14174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:07:00.156968 14188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:00.158402 14174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 11:07:00.158690 14174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d50de408c88141dc8e6d060b7aa0558a"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "9efab42c253d44c7e1482838be587960"
server_key_iv: "d283115fc6d98f410741aeef0b75b5b7"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.158923 14174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:00.175011 14174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:00.175917 14174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:00.176250 14174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:00.195013 14174 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34539
I20260925 11:07:00.195039 14239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34539 every 8 connection(s)
I20260925 11:07:00.196450 14174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:00.200279 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:00.200942 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14174
I20260925 11:07:00.201141 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:00.201977 8916 external_mini_cluster.cc:1428] Setting key b4d09e060f176eedcb6202129472534a
I20260925 11:07:00.214655 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a: Bootstrap starting.
I20260925 11:07:00.216264 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:00.203966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.8.181.62:34539)
0925 11:07:00.205139 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:00.205150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:00.205203 (+ 53us) server_negotiation.cc:407] Connection header received
0925 11:07:00.206925 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:00.206949 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:00.207427 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:00.207773 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:00.208388 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.209716 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.211645 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.212374 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.212510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:00.213783 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:00.213790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:00.213804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:00.214369 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:00.214447 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:00.214518 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:00.214618 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:00.214919 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 11:07:00.215275 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":537,"threads_started":1}
I20260925 11:07:00.218016 14240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:00.219177 14240 log.cc:824] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:00.220646 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:07:00.222169 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a: No bootstrap required, opened a new log
I20260925 11:07:00.230134 14240 raft_consensus.cc:348] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } }
I20260925 11:07:00.230607 14240 raft_consensus.cc:374] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:00.230731 14240 raft_consensus.cc:729] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d50de408c88141dc8e6d060b7aa0558a, State: Initialized, Role: FOLLOWER
I20260925 11:07:00.231387 14240 consensus_queue.cc:260] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } }
I20260925 11:07:00.231688 14240 raft_consensus.cc:388] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:00.231828 14240 raft_consensus.cc:482] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:00.231956 14240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:00.234063 14240 raft_consensus.cc:504] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } }
I20260925 11:07:00.234603 14240 leader_election.cc:302] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d50de408c88141dc8e6d060b7aa0558a; no voters:
I20260925 11:07:00.235194 14240 leader_election.cc:288] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:00.235379 14245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:00.236924 14245 raft_consensus.cc:686] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [term 1 LEADER]: Becoming Leader. State: Replica: d50de408c88141dc8e6d060b7aa0558a, State: Running, Role: LEADER
I20260925 11:07:00.237509 14240 sys_catalog.cc:564] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:00.237553 14245 consensus_queue.cc:237] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } }
I20260925 11:07:00.241074 14247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d50de408c88141dc8e6d060b7aa0558a. Latest consensus state: current_term: 1 leader_uuid: "d50de408c88141dc8e6d060b7aa0558a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } } }
I20260925 11:07:00.241138 14246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d50de408c88141dc8e6d060b7aa0558a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d50de408c88141dc8e6d060b7aa0558a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34539 } } }
I20260925 11:07:00.241504 14247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:00.241557 14246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:00.243227 14255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:00.249192 14255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:00.258693 14255 catalog_manager.cc:1347] Generated new cluster ID: b43d38de7e0345249edc0dc7c413520d
I20260925 11:07:00.258813 14255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:00.290678 14255 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:00.292145 14255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:00.301611 14255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d50de408c88141dc8e6d060b7aa0558a: Generated new TSK 0
I20260925 11:07:00.302625 14255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 11:07:00.349845 14270 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 11:07:00.349983 14270 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 11:07:00.351199 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:00.329261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.8.181.62:34539)
0925 11:07:00.329466 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:00.329473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:00.329953 (+ 480us) server_negotiation.cc:407] Connection header received
0925 11:07:00.330183 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:00.330190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:00.330294 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:00.330531 (+ 237us) server_negotiation.cc:226] Negotiated authn=JWT
0925 11:07:00.330984 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.332131 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.333544 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.334245 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.334384 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:00.335105 (+ 721us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 11:07:00.350387 (+ 15282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 11:07:00.350865 (+ 478us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 11:07:00.351336 14243 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 11:07:00.351404 14269 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:00.329103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34539 (local address 127.0.0.1:58410)
0925 11:07:00.329837 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:00.329859 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:00.329990 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:00.330550 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:00.330553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:00.330563 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 11:07:00.330826 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:00.330834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.332288 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.332292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:00.333398 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:00.333409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.334562 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.334569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:00.334727 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:00.334767 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 11:07:00.350758 (+ 15991us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 11:07:00.351010 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.181.62:34539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":404,"threads_started":1}
I20260925 11:07:00.353332 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14174
I20260925 11:07:00.354660 14235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:00.451633 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14174
2026-09-25T11:07:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (764 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 11:07:00.505314 8916 mini_oidc.cc:152] Starting JWKS server
I20260925 11:07:00.507453 8916 webserver.cc:460] Webserver started at https://127.0.0.1:35917/ using document root <none> and password file <none>
I20260925 11:07:00.507766 8916 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 11:07:00.508467 8916 webserver.cc:460] Webserver started at http://127.0.0.1:35531/ using document root <none> and password file <none>
2026-09-25T11:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:00Z Disabled control of system clock
I20260925 11:07:00.540726 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34057
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:39199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34057
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35917/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 11:07:00.883253 14277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:00.883567 14277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:00.883653 14277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:00.890976 14277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:00.891093 14277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:00.891129 14277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:00.891177 14277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:00.891219 14277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 11:07:00.891273 14277 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 11:07:00.891311 14277 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35917/jwks.json
I20260925 11:07:00.900945 14277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35917/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:00.902251 14277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:00.904068 14277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:00.916666 14285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:00.916772 14283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:00.916846 14282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:00.917470 14277 server_base.cc:1034] running on GCE node
I20260925 11:07:00.918373 14277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:00.919948 14277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:00.921164 14277 hybrid_clock.cc:648] HybridClock initialized: now 1790334420921120 us; error 42 us; skew 500 ppm
I20260925 11:07:00.921564 14277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:00.924081 14277 webserver.cc:460] Webserver started at http://127.8.181.62:45675/ using document root <none> and password file <none>
I20260925 11:07:00.924823 14277 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:00.924933 14277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:00.925292 14277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:00.928376 14277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "7f1a64f29f3a4b01b0faba045147e28b"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "fcbaf5b977e0ade10ee672747407c9a1"
server_key_iv: "90965e3bf1edc065918c94ff2c9197f3"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.929229 14277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "7f1a64f29f3a4b01b0faba045147e28b"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "fcbaf5b977e0ade10ee672747407c9a1"
server_key_iv: "90965e3bf1edc065918c94ff2c9197f3"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.935132 14277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 11:07:00.939229 14291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:00.940866 14277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:07:00.941105 14277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "7f1a64f29f3a4b01b0faba045147e28b"
format_stamp: "Formatted at 2026-09-25 11:07:00 on dist-test-slave-6042"
server_key: "fcbaf5b977e0ade10ee672747407c9a1"
server_key_iv: "90965e3bf1edc065918c94ff2c9197f3"
server_key_version: "encryptionkey@0"
I20260925 11:07:00.941365 14277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:00.958734 14277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:00.959630 14277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:00.959959 14277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:00.979305 14277 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34057
I20260925 11:07:00.979313 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34057 every 8 connection(s)
I20260925 11:07:00.980744 14277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:00.982383 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14277
I20260925 11:07:00.982722 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:00.983345 8916 external_mini_cluster.cc:1428] Setting key d690df935dca87cb24cc585e5e2de38b
I20260925 11:07:00.985430 14343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:00.997466 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:00.985452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.8.181.62:34057)
0925 11:07:00.986455 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:00.986599 (+ 144us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:00.986661 (+ 62us) server_negotiation.cc:407] Connection header received
0925 11:07:00.988314 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:00.988338 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:00.988854 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:00.989297 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:00.989954 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.991485 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.992858 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:00.993548 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:00.993747 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:00.995124 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:00.995132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:00.995148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:00.995748 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:00.995826 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:00.995849 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:00.995976 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:00.996210 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 11:07:00.996537 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":346,"threads_started":1}
I20260925 11:07:00.997570 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b: Bootstrap starting.
I20260925 11:07:01.000957 14343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:01.001379 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:07:01.002036 14343 log.cc:824] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:01.005466 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b: No bootstrap required, opened a new log
I20260925 11:07:01.013685 14343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } }
I20260925 11:07:01.014135 14343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:01.014256 14343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1a64f29f3a4b01b0faba045147e28b, State: Initialized, Role: FOLLOWER
I20260925 11:07:01.014982 14343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } }
I20260925 11:07:01.015236 14343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:01.015393 14343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:01.015582 14343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:01.017822 14343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } }
I20260925 11:07:01.018380 14343 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1a64f29f3a4b01b0faba045147e28b; no voters:
I20260925 11:07:01.019202 14343 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:01.019323 14348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:01.020489 14348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [term 1 LEADER]: Becoming Leader. State: Replica: 7f1a64f29f3a4b01b0faba045147e28b, State: Running, Role: LEADER
I20260925 11:07:01.021061 14348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } }
I20260925 11:07:01.021556 14343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:01.024070 14349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1a64f29f3a4b01b0faba045147e28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } } }
I20260925 11:07:01.024024 14350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1a64f29f3a4b01b0faba045147e28b. Latest consensus state: current_term: 1 leader_uuid: "7f1a64f29f3a4b01b0faba045147e28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1a64f29f3a4b01b0faba045147e28b" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34057 } } }
I20260925 11:07:01.024504 14349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:01.024500 14350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:01.025385 14354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:01.031015 14354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:01.041172 14354 catalog_manager.cc:1347] Generated new cluster ID: 9eed6a6e48bc4739a98f73c7a8261958
I20260925 11:07:01.041518 14354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:01.069757 14354 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:01.071769 14354 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:01.079084 14354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1a64f29f3a4b01b0faba045147e28b: Generated new TSK 0
I20260925 11:07:01.080147 14354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:07:01.100513 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:01.092540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.8.181.62:34057)
0925 11:07:01.092760 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:01.092768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:01.093140 (+ 372us) server_negotiation.cc:407] Connection header received
0925 11:07:01.093391 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:01.093395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:01.093480 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:01.093650 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0925 11:07:01.094108 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.094915 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.096312 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.096766 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.096912 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:01.097411 (+ 499us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 11:07:01.099850 (+ 2439us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 11:07:01.100201 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 11:07:01.100644 14346 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 11:07:01.102650 14372 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:01.092442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:34057 (local address 127.0.0.1:54618)
0925 11:07:01.093030 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:01.093050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:01.093216 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:01.093672 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:01.093677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:01.093689 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 11:07:01.093962 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:01.093971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.095058 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.095064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:01.096174 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:01.096185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.096968 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.096978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:01.097118 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:01.097157 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 11:07:01.100082 (+ 2925us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 11:07:01.102131 (+ 2049us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.181.62:34057: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1714,"thread_start_us":344,"threads_started":1}
I20260925 11:07:01.104681 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14277
I20260925 11:07:01.106036 14338 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 11:07:01.113022 14373 webserver.cc:533] Webserver: sslize failed:
E20260925 11:07:01.113135 14373 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 11:07:01.200344 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14277
2026-09-25T11:07:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T11:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:01Z Disabled control of system clock
I20260925 11:07:01.286553 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35147
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:35147
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/testchainca.cert with env {}
W20260925 11:07:01.642771 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:01.643088 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:01.643249 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:01.651180 14378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:01.651319 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:01.651412 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:01.651507 14378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:01.651598 14378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 11:07:01.651712 14378 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 11:07:01.651787 14378 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 11:07:01.662837 14378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:35147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:01.664294 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:01.665712 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:01.679042 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:01.679095 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:01.679119 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:01.680827 14378 server_base.cc:1034] running on GCE node
I20260925 11:07:01.681795 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:01.683518 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:01.684764 14378 hybrid_clock.cc:648] HybridClock initialized: now 1790334421684739 us; error 60 us; skew 500 ppm
I20260925 11:07:01.685205 14378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:01.694001 14378 webserver.cc:460] Webserver started at http://127.8.181.62:43669/ using document root <none> and password file <none>
I20260925 11:07:01.694998 14378 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:01.695102 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:01.695498 14378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:01.698750 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0"
format_stamp: "Formatted at 2026-09-25 11:07:01 on dist-test-slave-6042"
server_key: "54b7dbf8168bff04e0cad6ad24171483"
server_key_iv: "05335f6a88d7ef678201f6fe03c5a487"
server_key_version: "encryptionkey@0"
I20260925 11:07:01.699647 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0"
format_stamp: "Formatted at 2026-09-25 11:07:01 on dist-test-slave-6042"
server_key: "54b7dbf8168bff04e0cad6ad24171483"
server_key_iv: "05335f6a88d7ef678201f6fe03c5a487"
server_key_version: "encryptionkey@0"
I20260925 11:07:01.705242 14378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:07:01.709121 14392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:01.711023 14378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:07:01.711252 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0"
format_stamp: "Formatted at 2026-09-25 11:07:01 on dist-test-slave-6042"
server_key: "54b7dbf8168bff04e0cad6ad24171483"
server_key_iv: "05335f6a88d7ef678201f6fe03c5a487"
server_key_version: "encryptionkey@0"
I20260925 11:07:01.711534 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:01.730171 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:01.731182 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:01.731498 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:01.753008 14378 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:35147
I20260925 11:07:01.753111 14443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:35147 every 8 connection(s)
I20260925 11:07:01.754436 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:01.758592 14444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:01.760244 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14378
I20260925 11:07:01.760475 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:01.761155 8916 external_mini_cluster.cc:1428] Setting key 7e9df1d23ca1d52ecae0fc870e3d3ea9
I20260925 11:07:01.773108 14444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0: Bootstrap starting.
I20260925 11:07:01.777508 14444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:01.778255 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:01.763287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.8.181.62:35147)
0925 11:07:01.764565 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:01.764587 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:01.764648 (+ 61us) server_negotiation.cc:407] Connection header received
0925 11:07:01.767120 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:01.767153 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:01.767786 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:01.768270 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:01.768949 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.771120 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.772497 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.773587 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.773781 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:01.775092 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:01.775104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:01.775137 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:01.775896 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:01.776017 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:01.776054 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:01.776195 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:01.776463 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 11:07:01.776829 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":408,"threads_started":1}
I20260925 11:07:01.778877 14444 log.cc:824] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:01.782315 14444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0: No bootstrap required, opened a new log
I20260925 11:07:01.783041 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:07:01.790798 14444 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } }
I20260925 11:07:01.791379 14444 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:01.791501 14444 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69fb8cd02ad4cbbbddb8ba50eb98dd0, State: Initialized, Role: FOLLOWER
I20260925 11:07:01.792073 14444 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } }
I20260925 11:07:01.792338 14444 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:01.792440 14444 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:01.792562 14444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:01.794852 14444 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } }
I20260925 11:07:01.795365 14444 leader_election.cc:302] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69fb8cd02ad4cbbbddb8ba50eb98dd0; no voters:
I20260925 11:07:01.796033 14444 leader_election.cc:288] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:01.796226 14449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:01.797358 14449 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [term 1 LEADER]: Becoming Leader. State: Replica: e69fb8cd02ad4cbbbddb8ba50eb98dd0, State: Running, Role: LEADER
I20260925 11:07:01.797960 14449 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } }
I20260925 11:07:01.798565 14444 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:01.800930 14450 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } } }
I20260925 11:07:01.801039 14451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69fb8cd02ad4cbbbddb8ba50eb98dd0. Latest consensus state: current_term: 1 leader_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69fb8cd02ad4cbbbddb8ba50eb98dd0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35147 } } }
I20260925 11:07:01.801359 14450 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:01.801466 14451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:01.805430 14456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:01.811036 14456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:01.820322 14456 catalog_manager.cc:1347] Generated new cluster ID: e9168049c1904876b85a24d7d5a44c7b
I20260925 11:07:01.820468 14456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:01.838460 14456 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:01.839942 14456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:01.847579 14456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69fb8cd02ad4cbbbddb8ba50eb98dd0: Generated new TSK 0
I20260925 11:07:01.848629 14456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:07:02.388341 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:01.871076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34774 (local address 127.8.181.62:35147)
0925 11:07:01.871291 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:01.871299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:01.871768 (+ 469us) server_negotiation.cc:407] Connection header received
0925 11:07:01.872054 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:01.872059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:01.872136 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:01.872305 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0925 11:07:01.872810 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.873540 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.875012 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.875490 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.875644 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:01.876170 (+ 526us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 11:07:02.387478 (+511308us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 11:07:02.388031 (+ 553us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 11:07:02.388478 14447 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34774: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 11:07:02.388669 14473 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:01.870967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35147 (local address 127.0.0.1:34774)
0925 11:07:01.871654 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:01.871671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:01.871858 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:01.872320 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:01.872325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:01.872336 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0925 11:07:01.872653 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:01.872661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.873703 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.873708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:01.874863 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:01.874875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:01.875695 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:01.875702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:01.875861 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:01.875899 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 11:07:02.387887 (+511988us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 11:07:02.388227 (+ 340us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.181.62:35147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":385,"threads_started":1}
I20260925 11:07:02.390044 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14378
I20260925 11:07:02.391352 14439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:02.495855 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14378
2026-09-25T11:07:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1278 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (448 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (441 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:03 dist-test-slave-6042 krb5kdc[14488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:03 dist-test-slave-6042 krb5kdc[14488](info): set up 2 sockets
Sep 25 11:07:03 dist-test-slave-6042 krb5kdc[14488](info): commencing operation
krb5kdc: starting...
W20260925 11:07:04.519754 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:04 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:07:04.764130 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37407
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:05.102355 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:05.102782 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:05.102874 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:05.110610 14501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:05.110740 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:05.110791 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:05.110841 14501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:05.110888 14501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:05.121208 14501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:37407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:05.122645 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:05.124025 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:05.137036 14507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:05.137125 14506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:05.137125 14509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:05.138280 14501 server_base.cc:1034] running on GCE node
I20260925 11:07:05.139240 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:05.140967 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:05.142210 14501 hybrid_clock.cc:648] HybridClock initialized: now 1790334425142165 us; error 38 us; skew 500 ppm
I20260925 11:07:05.142689 14501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:05 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334425, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:05.146677 14501 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:07:05.148316 14501 webserver.cc:460] Webserver started at http://127.8.181.62:37701/ using document root <none> and password file <none>
I20260925 11:07:05.149070 14501 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:05.149226 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:05.149562 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:05.152745 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "afbf63c917af49f5b6b3498f30f2de88"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "3a8ba16a56333e0e72b67b3db61ef7fb"
server_key_iv: "d8aff634f05f9b6455c18e9bfaf4b45b"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.153600 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "afbf63c917af49f5b6b3498f30f2de88"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "3a8ba16a56333e0e72b67b3db61ef7fb"
server_key_iv: "d8aff634f05f9b6455c18e9bfaf4b45b"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.159484 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 11:07:05.164211 14516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:05.166102 14501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:05.166374 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "afbf63c917af49f5b6b3498f30f2de88"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "3a8ba16a56333e0e72b67b3db61ef7fb"
server_key_iv: "d8aff634f05f9b6455c18e9bfaf4b45b"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.166752 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:05.182771 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:05.187803 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:05.188135 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:05.208353 14501 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:37407
I20260925 11:07:05.208392 14567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:37407 every 8 connection(s)
I20260925 11:07:05.209858 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:05.213958 14568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:05.217208 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14501
I20260925 11:07:05.217425 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:05.218096 8916 external_mini_cluster.cc:1428] Setting key 10a18b407c191424589c51179c34ddd1
I20260925 11:07:05.228050 14568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88: Bootstrap starting.
I20260925 11:07:05.232270 14568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:05.233783 14568 log.cc:824] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:07:05 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:05.236936 14568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88: No bootstrap required, opened a new log
I20260925 11:07:05.243636 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:05.220319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.8.181.62:37407)
0925 11:07:05.221159 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:05.221170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:05.221213 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:07:05.223073 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:05.223099 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:05.223566 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:05.223985 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:05.225865 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.227524 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.229352 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.230166 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.230323 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:05.235482 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:05.235491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:05.235508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:05.235547 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:05.239070 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:05.239921 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:05.239930 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:05.239938 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:05.240130 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:05.240558 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:05.240567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:05.240571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:05.241178 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:05.241544 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:05.241970 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 11:07:05.242453 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":408,"threads_started":1}
I20260925 11:07:05.245419 14568 raft_consensus.cc:348] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } }
I20260925 11:07:05.245851 14568 raft_consensus.cc:374] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:05.245977 14568 raft_consensus.cc:729] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf63c917af49f5b6b3498f30f2de88, State: Initialized, Role: FOLLOWER
I20260925 11:07:05.246797 14568 consensus_queue.cc:260] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } }
I20260925 11:07:05.247097 14568 raft_consensus.cc:388] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:05.247262 14568 raft_consensus.cc:482] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:05.247452 14568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:05.249682 14568 raft_consensus.cc:504] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } }
I20260925 11:07:05.250242 14568 leader_election.cc:302] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afbf63c917af49f5b6b3498f30f2de88; no voters:
I20260925 11:07:05.250962 14568 leader_election.cc:288] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:05.251143 14573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:05.251950 14573 raft_consensus.cc:686] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [term 1 LEADER]: Becoming Leader. State: Replica: afbf63c917af49f5b6b3498f30f2de88, State: Running, Role: LEADER
I20260925 11:07:05.252575 14573 consensus_queue.cc:237] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } }
I20260925 11:07:05.253456 14568 sys_catalog.cc:564] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:05.255561 14575 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afbf63c917af49f5b6b3498f30f2de88. Latest consensus state: current_term: 1 leader_uuid: "afbf63c917af49f5b6b3498f30f2de88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } } }
I20260925 11:07:05.256090 14575 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:05.256702 14574 sys_catalog.cc:455] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afbf63c917af49f5b6b3498f30f2de88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afbf63c917af49f5b6b3498f30f2de88" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 37407 } } }
I20260925 11:07:05.257223 14574 sys_catalog.cc:458] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:05.270130 14582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:05.276057 14582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:05.285955 14582 catalog_manager.cc:1347] Generated new cluster ID: c1914cea6e6a445e96b50b5ef7371810
I20260925 11:07:05.286156 14582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:05.302731 14582 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:05.304719 14582 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:05.316066 14582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afbf63c917af49f5b6b3498f30f2de88: Generated new TSK 0
I20260925 11:07:05.317248 14582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:07:05.384496 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:05.741570 14595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:05.741879 14595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:05.742044 14595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:05.749989 14595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:05.750120 14595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:05.750314 14595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:05.761698 14595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:05.763375 14595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:05.764818 14595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:05.779366 14600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:05.779390 14601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:05.779587 14603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:05.780333 14595 server_base.cc:1034] running on GCE node
I20260925 11:07:05.780958 14595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:05.782058 14595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:05.783308 14595 hybrid_clock.cc:648] HybridClock initialized: now 1790334425783266 us; error 35 us; skew 500 ppm
I20260925 11:07:05.783664 14595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:05 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334425, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:05.787472 14595 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:07:05.789175 14595 webserver.cc:460] Webserver started at http://127.8.181.1:36841/ using document root <none> and password file <none>
I20260925 11:07:05.789892 14595 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:05.790007 14595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:05.790266 14595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:05.793399 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "8573b27d1b334fdc8324b26551b001fc"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "ef5a589d7c68d3c7664bee5a54c1089e"
server_key_iv: "ba8aaf275e560d5adadecb2113b70404"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.794176 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "8573b27d1b334fdc8324b26551b001fc"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "ef5a589d7c68d3c7664bee5a54c1089e"
server_key_iv: "ba8aaf275e560d5adadecb2113b70404"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.799772 14595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 11:07:05.803625 14610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:05.805246 14595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:07:05.805402 14595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "8573b27d1b334fdc8324b26551b001fc"
format_stamp: "Formatted at 2026-09-25 11:07:05 on dist-test-slave-6042"
server_key: "ef5a589d7c68d3c7664bee5a54c1089e"
server_key_iv: "ba8aaf275e560d5adadecb2113b70404"
server_key_version: "encryptionkey@0"
I20260925 11:07:05.805616 14595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:05.827688 14595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:05.832713 14595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:05.833053 14595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:05.834131 14595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:05.835628 14595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:05.835754 14595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:05.835911 14595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:05.835986 14595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:05.872113 14595 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:45305
I20260925 11:07:05.872133 14722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:45305 every 8 connection(s)
I20260925 11:07:05.873556 14595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:05.879415 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14595
I20260925 11:07:05.879660 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:05.880337 8916 external_mini_cluster.cc:1428] Setting key c57072b75642f9ed4c61c4707eeb22b4
Sep 25 11:07:05 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334425, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:05.896502 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:05.876812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:53839 (local address 127.8.181.62:37407)
0925 11:07:05.877325 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:05.877330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:05.878383 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 11:07:05.880649 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:05.880654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:05.880752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:05.880892 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:05.882866 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.883609 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.885227 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.885654 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.885811 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:05.890107 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:05.890113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:05.890116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:05.890157 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:05.893110 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:05.893974 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:05.893984 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:05.893986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:05.894104 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:05.894673 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:05.894678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:05.894680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:05.894955 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:05.895125 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:05.895879 (+ 754us) server_negotiation.cc:299] Negotiation successful
0925 11:07:05.896133 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":283,"threads_started":1}
I20260925 11:07:05.897607 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:05.877275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37407 (local address 127.8.181.1:53839)
0925 11:07:05.878191 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:05.878237 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:05.880273 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:05.881177 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:05.881186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:05.881877 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:05.882683 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:05.882704 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.883786 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.883791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:05.885064 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:05.885077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:05.885907 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:05.885916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:05.886176 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:05.887074 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:05.887100 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:05.889805 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:05.893306 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:05.893317 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:05.893338 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:05.893819 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:05.894226 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:05.894231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:05.894235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:05.894431 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:05.895095 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:05.895104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:05.895484 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 11:07:05.895764 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 11:07:05.896357 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":379,"threads_started":1}
I20260925 11:07:05.899598 14723 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37407
I20260925 11:07:05.899983 14723 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:07:05.901048 14723 heartbeater.cc:511] Master 127.8.181.62:37407 requested a full tablet report, sending...
I20260925 11:07:05.903802 14533 ts_manager.cc:194] Registered new tserver with Master: 8573b27d1b334fdc8324b26551b001fc (127.8.181.1:45305)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:07:05.905412 14533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:53839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:07:05.936671 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:06.291383 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:06.291736 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:06.291940 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:06.300772 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:06.300922 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:06.301147 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:06.313040 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:06.314662 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:06.316361 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:06.332376 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:06.332360 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:06.333083 14734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:06.333284 14729 server_base.cc:1034] running on GCE node
I20260925 11:07:06.333954 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:06.335151 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:06.336377 14729 hybrid_clock.cc:648] HybridClock initialized: now 1790334426336322 us; error 61 us; skew 500 ppm
I20260925 11:07:06.336690 14729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:06 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334426, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:06.340375 14729 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:07:06.342155 14729 webserver.cc:460] Webserver started at http://127.8.181.2:35123/ using document root <none> and password file <none>
I20260925 11:07:06.342947 14729 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:06.343081 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:06.343423 14729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:06.346990 14729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "dd564bfb72814e5ca813b0dbbae13271"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "2bfc5725b5386e3c0c4fb4674ff9788a"
server_key_iv: "0802f3e4be9d079dd0849990a68439f1"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.347954 14729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "dd564bfb72814e5ca813b0dbbae13271"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "2bfc5725b5386e3c0c4fb4674ff9788a"
server_key_iv: "0802f3e4be9d079dd0849990a68439f1"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.354784 14729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:07:06.359242 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.361234 14729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:06.361546 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "dd564bfb72814e5ca813b0dbbae13271"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "2bfc5725b5386e3c0c4fb4674ff9788a"
server_key_iv: "0802f3e4be9d079dd0849990a68439f1"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.361860 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:06.376466 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:06.381948 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:06.382299 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:06.383440 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:06.385125 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:06.385210 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.385356 14729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:06.385409 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.422751 14729 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44523
I20260925 11:07:06.422803 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44523 every 8 connection(s)
I20260925 11:07:06.424263 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:06.431226 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14729
I20260925 11:07:06.431442 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:06.432066 8916 external_mini_cluster.cc:1428] Setting key 01d67d0f9f12441626659e4d65d352a0
Sep 25 11:07:06 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334426, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:06.448611 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:06.427400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:47441 (local address 127.8.181.62:37407)
0925 11:07:06.428018 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:06.428024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:06.428969 (+ 945us) server_negotiation.cc:407] Connection header received
0925 11:07:06.431037 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:06.431043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:06.431146 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:06.431282 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:06.433173 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.433945 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.435705 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.436149 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.436308 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:06.440341 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:06.440347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:06.440350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:06.440389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:06.444421 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:06.445599 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:06.445608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:06.445615 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:06.445794 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:06.446314 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:06.446322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:06.446328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:06.446789 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:06.446956 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:06.447794 (+ 838us) server_negotiation.cc:299] Negotiation successful
0925 11:07:06.448090 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":391,"threads_started":1}
I20260925 11:07:06.449421 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:06.427918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37407 (local address 127.8.181.2:47441)
0925 11:07:06.428776 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:06.428819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:06.430714 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:06.431551 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:06.431563 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:06.432166 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:06.432977 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:06.433001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.434130 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.434141 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:06.435522 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:06.435535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.436418 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.436426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:06.436681 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:06.437453 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:06.437479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:06.439961 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:06.444802 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:06.444812 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:06.444828 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:06.445400 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:06.445934 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:06.445937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:06.445940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:06.446163 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:06.446936 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:06.446943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:06.447362 (+ 419us) client_negotiation.cc:769] Sending connection context
0925 11:07:06.447630 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 11:07:06.448088 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":366,"threads_started":1}
I20260925 11:07:06.451314 14857 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37407
I20260925 11:07:06.451707 14857 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:06.452517 14857 heartbeater.cc:511] Master 127.8.181.62:37407 requested a full tablet report, sending...
I20260925 11:07:06.454315 14533 ts_manager.cc:194] Registered new tserver with Master: dd564bfb72814e5ca813b0dbbae13271 (127.8.181.2:44523)
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:07:06.455490 14533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:47441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:07:06.488066 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:06.846807 14863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:06.847188 14863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:06.847374 14863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:06.855329 14863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:06.855505 14863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:06.855684 14863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:06.866861 14863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:37407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:06.868381 14863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:06.870240 14863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:06.885824 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:06.885797 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:06.885859 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:06.886747 14863 server_base.cc:1034] running on GCE node
I20260925 11:07:06.887467 14863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:06.888613 14863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:06.889915 14863 hybrid_clock.cc:648] HybridClock initialized: now 1790334426889826 us; error 57 us; skew 500 ppm
I20260925 11:07:06.890340 14863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:06 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334426, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:06.898914 14863 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:07:06.900933 14863 webserver.cc:460] Webserver started at http://127.8.181.3:46847/ using document root <none> and password file <none>
I20260925 11:07:06.901801 14863 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:06.901925 14863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:06.902323 14863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:06.905579 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "f64c1e230316463c801582adda8c677b"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "8096a4d644aa3c02a7b85cc75c3400ec"
server_key_iv: "147bedd3c6e49263ec1d6b051eb1d97d"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.906726 14863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "f64c1e230316463c801582adda8c677b"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "8096a4d644aa3c02a7b85cc75c3400ec"
server_key_iv: "147bedd3c6e49263ec1d6b051eb1d97d"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.908829 14723 heartbeater.cc:503] Master 127.8.181.62:37407 was elected leader, sending a full tablet report...
I20260925 11:07:06.913051 14863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:07:06.917280 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.919175 14863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:06.919405 14863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "f64c1e230316463c801582adda8c677b"
format_stamp: "Formatted at 2026-09-25 11:07:06 on dist-test-slave-6042"
server_key: "8096a4d644aa3c02a7b85cc75c3400ec"
server_key_iv: "147bedd3c6e49263ec1d6b051eb1d97d"
server_key_version: "encryptionkey@0"
I20260925 11:07:06.919667 14863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:06.937561 14863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:06.942363 14863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:06.942718 14863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:06.943833 14863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:06.945217 14863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:06.945293 14863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.945387 14863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:06.945434 14863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:06.984143 14863 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:40033
I20260925 11:07:06.984252 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:40033 every 8 connection(s)
I20260925 11:07:06.985538 14863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:06.992904 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 14863
I20260925 11:07:06.993142 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:06.993649 8916 external_mini_cluster.cc:1428] Setting key aabc8efc6e8016288d9276ed761e2ac6
Sep 25 11:07:06 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334426, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:07.010728 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:06.988565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:41153 (local address 127.8.181.62:37407)
0925 11:07:06.989172 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:06.989177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:06.990092 (+ 915us) server_negotiation.cc:407] Connection header received
0925 11:07:06.992269 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:06.992275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:06.992401 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:06.992557 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:06.994456 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.995515 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.997981 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.998440 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.998703 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.002993 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.002999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.003003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.003044 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.006288 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.007375 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.007380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.007383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.007564 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.008263 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.008308 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.008314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.008794 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.009009 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.009913 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.010188 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":281,"threads_started":1}
I20260925 11:07:07.011878 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:06.989015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:37407 (local address 127.8.181.3:41153)
0925 11:07:06.989876 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:06.989921 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:06.991840 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:06.992822 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:06.992831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:06.993506 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:06.994215 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:06.994236 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.995883 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.995889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:06.997796 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:06.997811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:06.998853 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:06.998859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:06.999106 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:06.999869 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:06.999895 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:07.002674 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:07.006640 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:07.006649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:07.006667 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:07.007214 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:07.007779 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:07.007783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:07.007786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:07.008075 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:07.008990 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:07.009000 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:07.009472 (+ 472us) client_negotiation.cc:769] Sending connection context
0925 11:07:07.009804 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 11:07:07.010572 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":333,"threads_started":1}
I20260925 11:07:07.013749 14991 heartbeater.cc:347] Connected to a master server at 127.8.181.62:37407
I20260925 11:07:07.014099 14991 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:07.014995 14991 heartbeater.cc:511] Master 127.8.181.62:37407 requested a full tablet report, sending...
I20260925 11:07:07.016687 14533 ts_manager.cc:194] Registered new tserver with Master: f64c1e230316463c801582adda8c677b (127.8.181.3:40033)
I20260925 11:07:07.017632 14533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:41153
I20260925 11:07:07.023770 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/test-admin.keytab.
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.054368 8916 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 11:07:07.060639 14999 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:07.057822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.8.181.1:45305)
0925 11:07:07.058611 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.058619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.058656 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:07:07.058723 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.058729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.058950 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.059207 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.060239 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50194: BlockingRecv error: recv got EOF from 127.0.0.1:50194 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":384,"threads_started":1}
W20260925 11:07:07.061045 8916 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc' of type FILE
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.063223 8916 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.065090 8916 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:07:07.083927 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.067356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.8.181.1:45305)
0925 11:07:07.067750 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.067757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.067945 (+ 188us) server_negotiation.cc:407] Connection header received
0925 11:07:07.068195 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.068200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.068265 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.068374 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.069506 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.070582 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.071901 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.072487 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.072702 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.076942 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.076957 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.076974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.077022 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.080274 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.081257 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.081265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.081268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.081413 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.081965 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.081969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.081971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.082430 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.082821 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.083356 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.083571 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 11:07:07.091079 14999 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:07.088958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.8.181.1:45305)
0925 11:07:07.089221 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.089230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.089588 (+ 358us) server_negotiation.cc:407] Connection header received
0925 11:07:07.089805 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.089808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.089868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.089978 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.090829 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50198: BlockingRecv error: recv got EOF from 127.0.0.1:50198 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260925 11:07:07.091447 8916 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc' of type FILE
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.093281 8916 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.094887 8916 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:07:07.112089 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.096532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.8.181.1:45305)
0925 11:07:07.096870 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.096878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.097156 (+ 278us) server_negotiation.cc:407] Connection header received
0925 11:07:07.097420 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.097424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.097492 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.097609 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.098796 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.099591 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.100967 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.101471 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.101634 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.106219 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.106234 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.106238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.106302 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.109138 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.110038 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.110044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.110048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.110191 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.110699 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.110710 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.110712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.110983 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.111279 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.111570 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.111788 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260925 11:07:07.117206 14999 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:07.115185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.8.181.1:45305)
0925 11:07:07.115416 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.115423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.115670 (+ 247us) server_negotiation.cc:407] Connection header received
0925 11:07:07.115885 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.115891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.115972 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.116096 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.116956 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50208: BlockingRecv error: recv got EOF from 127.0.0.1:50208 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 11:07:07.117506 8916 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790334376327139-8916-0/krb5kdc/krb5cc' of type FILE
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.119338 8916 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:07.120807 8916 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:07:07.137574 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.122320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.8.181.1:45305)
0925 11:07:07.122615 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.122621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.122920 (+ 299us) server_negotiation.cc:407] Connection header received
0925 11:07:07.123153 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.123164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.123233 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.123336 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.124455 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.125306 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.126655 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.127326 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.127442 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.131830 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.131836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.131845 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.131887 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.134794 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.135554 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.135557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.135560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.135801 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.136212 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.136220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.136223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.136526 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.136764 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.137036 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.137222 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 11:07:07.139528 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14595
I20260925 11:07:07.153348 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.140455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.8.181.1:45305)
0925 11:07:07.140720 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.140725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.141130 (+ 405us) server_negotiation.cc:407] Connection header received
0925 11:07:07.141207 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.141210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.141270 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.141356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.142421 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.143279 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.144560 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.145027 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.145136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.147563 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.147569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.147572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.147611 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.150338 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.151209 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.151216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.151220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.151376 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.151787 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.151793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.151797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.152083 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.152414 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.152762 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.152996 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 11:07:07.154008 14718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:07.244567 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14595
I20260925 11:07:07.261893 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14729
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:07:07.279817 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.262938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.8.181.2:44523)
0925 11:07:07.263490 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.263498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.263527 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:07:07.263619 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.263633 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.263879 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.264031 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.265168 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.266438 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.268238 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.268817 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.268931 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.273326 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.273341 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.273358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.273406 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.276622 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.277522 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.277530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.277532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.277684 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.278101 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.278110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.278114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.278533 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.278864 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.279221 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.279486 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":306,"threads_started":1}
I20260925 11:07:07.281066 14852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:07.366175 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14729
I20260925 11:07:07.384042 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14863
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[14488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:07:07.402786 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:07.385197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.8.181.3:40033)
0925 11:07:07.385929 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:07.385941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:07.385980 (+ 39us) server_negotiation.cc:407] Connection header received
0925 11:07:07.386054 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:07.386062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:07.386328 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:07.386568 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:07.387716 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.388743 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.390138 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:07.390841 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:07.390992 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:07.395855 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:07.395871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:07.395888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:07.395930 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:07.399306 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.400255 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.400262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.400265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.400409 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:07.400875 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:07.400878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:07.400880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:07.401268 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:07.401505 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:07.402055 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 11:07:07.402312 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":399,"threads_started":1}
I20260925 11:07:07.404124 14986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:07.492760 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14863
I20260925 11:07:07.510715 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 14501
I20260925 11:07:07.512094 14563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:07.619517 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 14501
2026-09-25T11:07:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4239 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[15019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[15019](info): set up 2 sockets
Sep 25 11:07:07 dist-test-slave-6042 krb5kdc[15019](info): commencing operation
krb5kdc: starting...
W20260925 11:07:08.752672 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:08 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab.
Entry for principal oryx/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab.
I20260925 11:07:09.019785 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36487
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab
--principal=oryx/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:09.398577 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:09.399171 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:09.399366 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:09.407312 15032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:09.407464 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:09.407567 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:09.407922 15032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:09.408010 15032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:09.419204 15032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab
--principal=oryx/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:09.420984 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:09.422326 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:09.435712 15037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:09.435827 15040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:09.436264 15038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:09.437505 15032 server_base.cc:1034] running on GCE node
I20260925 11:07:09.438405 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:09.440138 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:09.441376 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790334429441321 us; error 53 us; skew 500 ppm
I20260925 11:07:09.441758 15032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:09 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334429, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:09.445456 15032 init.cc:377] Logged in from keytab as oryx/127.8.181.62@KRBTEST.COM (short username oryx)
I20260925 11:07:09.447454 15032 webserver.cc:460] Webserver started at http://127.8.181.62:40357/ using document root <none> and password file <none>
I20260925 11:07:09.448266 15032 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:09.448414 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:09.448770 15032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:09.452206 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "b71bfda482564e31996febbc4bcd4cc4"
format_stamp: "Formatted at 2026-09-25 11:07:09 on dist-test-slave-6042"
server_key: "9b1a2cc12aa1e63b4586d3b0f2268e1e"
server_key_iv: "91b4d5cf652ae539f3d07480b7690c77"
server_key_version: "encryptionkey@0"
I20260925 11:07:09.453078 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "b71bfda482564e31996febbc4bcd4cc4"
format_stamp: "Formatted at 2026-09-25 11:07:09 on dist-test-slave-6042"
server_key: "9b1a2cc12aa1e63b4586d3b0f2268e1e"
server_key_iv: "91b4d5cf652ae539f3d07480b7690c77"
server_key_version: "encryptionkey@0"
I20260925 11:07:09.459236 15032 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:07:09.463617 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:09.465910 15032 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260925 11:07:09.466178 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "b71bfda482564e31996febbc4bcd4cc4"
format_stamp: "Formatted at 2026-09-25 11:07:09 on dist-test-slave-6042"
server_key: "9b1a2cc12aa1e63b4586d3b0f2268e1e"
server_key_iv: "91b4d5cf652ae539f3d07480b7690c77"
server_key_version: "encryptionkey@0"
I20260925 11:07:09.466521 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:09.483265 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:09.488440 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:09.488829 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:09.511464 15032 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:36487
I20260925 11:07:09.511518 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:36487 every 8 connection(s)
I20260925 11:07:09.513206 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:09.517470 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:09.517910 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15032
I20260925 11:07:09.518146 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:09.519032 8916 external_mini_cluster.cc:1428] Setting key b13006eb008bcc116facf99ad80ca434
I20260925 11:07:09.531965 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Bootstrap starting.
Sep 25 11:07:09 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:09.535742 15099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:09.537552 15099 log.cc:824] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:09.541450 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: No bootstrap required, opened a new log
I20260925 11:07:09.545675 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:09.521527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.8.181.62:36487)
0925 11:07:09.522404 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:09.522418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:09.522635 (+ 217us) server_negotiation.cc:407] Connection header received
0925 11:07:09.524819 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:09.524847 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:09.525355 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:09.525818 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:09.527546 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:09.528933 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:09.530296 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:09.531127 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:09.531331 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:09.535811 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:09.535821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:09.535839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:09.535925 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:09.540343 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:09.541238 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:09.541248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:09.541257 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:09.541514 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:09.541973 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:09.541980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:09.541986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:09.542834 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:09.543195 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:09.543602 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 11:07:09.544132 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":391,"threads_started":1}
I20260925 11:07:09.551815 15099 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } }
I20260925 11:07:09.552220 15099 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:09.552290 15099 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71bfda482564e31996febbc4bcd4cc4, State: Initialized, Role: FOLLOWER
I20260925 11:07:09.552891 15099 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } }
I20260925 11:07:09.553115 15099 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:09.553210 15099 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:09.553339 15099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:09.555721 15099 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } }
I20260925 11:07:09.556237 15099 leader_election.cc:302] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71bfda482564e31996febbc4bcd4cc4; no voters:
I20260925 11:07:09.557009 15099 leader_election.cc:288] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:09.557184 15104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:09.557950 15104 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 LEADER]: Becoming Leader. State: Replica: b71bfda482564e31996febbc4bcd4cc4, State: Running, Role: LEADER
I20260925 11:07:09.558665 15104 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } }
I20260925 11:07:09.559765 15099 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:09.562080 15105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71bfda482564e31996febbc4bcd4cc4. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } }
I20260925 11:07:09.561985 15106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } }
I20260925 11:07:09.562521 15106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:09.562531 15105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:09.564921 15112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:09.571275 15112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:09.582763 15112 catalog_manager.cc:1347] Generated new cluster ID: 7075dcfbc4b14228bfe253f037ba1e62
I20260925 11:07:09.582903 15112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:09.598677 15112 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:09.600694 15112 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:09.620659 15112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Generated new TSK 0
I20260925 11:07:09.621943 15112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
I20260925 11:07:09.683696 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:10.044814 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:10.045228 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:10.045327 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:10.052974 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:10.053118 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:10.053241 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:10.063872 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:10.065357 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:10.066709 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:10.081602 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:10.081671 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:10.081673 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:10.082803 15125 server_base.cc:1034] running on GCE node
I20260925 11:07:10.083549 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:10.084797 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:10.086076 15125 hybrid_clock.cc:648] HybridClock initialized: now 1790334430085998 us; error 68 us; skew 500 ppm
I20260925 11:07:10.086436 15125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:10 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334430, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:10.090246 15125 init.cc:377] Logged in from keytab as oryx/127.8.181.1@KRBTEST.COM (short username oryx)
I20260925 11:07:10.091979 15125 webserver.cc:460] Webserver started at http://127.8.181.1:36973/ using document root <none> and password file <none>
I20260925 11:07:10.092808 15125 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:10.092986 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:10.093397 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:10.097254 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "c3e1d5abc8084c0ea417eae24c126818"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "d3e783599697bcc90e4aede68cbc0e89"
server_key_iv: "e8bfed269f962d6905534df1b3d68f58"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.098220 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "c3e1d5abc8084c0ea417eae24c126818"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "d3e783599697bcc90e4aede68cbc0e89"
server_key_iv: "e8bfed269f962d6905534df1b3d68f58"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.104466 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 11:07:10.108739 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.110754 15125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:10.111016 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "c3e1d5abc8084c0ea417eae24c126818"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "d3e783599697bcc90e4aede68cbc0e89"
server_key_iv: "e8bfed269f962d6905534df1b3d68f58"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.111294 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:10.124607 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:10.129693 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:10.130025 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:10.131273 15125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:10.132726 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:10.132838 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.132984 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:10.133059 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.168783 15125 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:40931
I20260925 11:07:10.168843 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:40931 every 8 connection(s)
I20260925 11:07:10.170186 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:10.178243 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15125
I20260925 11:07:10.178458 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:10.179147 8916 external_mini_cluster.cc:1428] Setting key f9cda973bcbd96e32460c7cca69624a3
Sep 25 11:07:10 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334430, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:10.194162 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:10.173101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:47001 (local address 127.8.181.62:36487)
0925 11:07:10.173592 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:10.173598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:10.174439 (+ 841us) server_negotiation.cc:407] Connection header received
0925 11:07:10.176573 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:10.176580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:10.176698 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:10.176835 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:10.178768 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.179624 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.181785 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.182214 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.182380 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:10.186239 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:10.186245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:10.186248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:10.186284 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:10.190181 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:10.191225 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:10.191234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:10.191241 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:10.191405 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:10.191888 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:10.191896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:10.191902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:10.192337 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:10.192508 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:10.193379 (+ 871us) server_negotiation.cc:299] Negotiation successful
0925 11:07:10.193669 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260925 11:07:10.195223 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:10.173471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:47001)
0925 11:07:10.174254 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:10.174295 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:10.176238 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:10.177118 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:10.177129 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:10.177756 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:10.178541 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:10.178579 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.179819 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.179825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:10.181598 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:10.181618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.182536 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.182544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:10.182798 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:10.183596 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:10.183620 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:10.185950 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:10.190393 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:10.190403 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:10.190420 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:10.191042 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:10.191537 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:10.191541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:10.191543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:10.191755 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:10.192480 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:10.192487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:10.192952 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 11:07:10.193197 (+ 245us) client_negotiation.cc:240] Negotiation successful
0925 11:07:10.193778 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":304,"threads_started":1}
I20260925 11:07:10.197190 15253 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:10.197584 15253 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:10.198658 15253 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:10.201455 15064 ts_manager.cc:194] Registered new tserver with Master: c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
WARNING: no policy specified for oryx/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:07:10.203202 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:47001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
I20260925 11:07:10.235556 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:10.595728 15259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:10.596046 15259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:10.596222 15259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:10.604807 15259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:10.604938 15259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:10.605134 15259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:10.616153 15259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:10.617708 15259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:10.619238 15259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:10.634743 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:10.634855 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:10.634829 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:10.635784 15259 server_base.cc:1034] running on GCE node
I20260925 11:07:10.636479 15259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:10.637617 15259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:10.638851 15259 hybrid_clock.cc:648] HybridClock initialized: now 1790334430638788 us; error 54 us; skew 500 ppm
I20260925 11:07:10.639197 15259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:10 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334430, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:10.642972 15259 init.cc:377] Logged in from keytab as oryx/127.8.181.2@KRBTEST.COM (short username oryx)
I20260925 11:07:10.644794 15259 webserver.cc:460] Webserver started at http://127.8.181.2:37099/ using document root <none> and password file <none>
I20260925 11:07:10.645524 15259 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:10.645629 15259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:10.645879 15259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:10.650064 15259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "b3e319f8099e415191485ba1ec096229"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "997f9911fe64e53ed2f0477187d2a396"
server_key_iv: "3f2a4a8d3fba1d3682c96f7c7397cb95"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.651203 15259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "b3e319f8099e415191485ba1ec096229"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "997f9911fe64e53ed2f0477187d2a396"
server_key_iv: "3f2a4a8d3fba1d3682c96f7c7397cb95"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.657375 15259 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:07:10.661569 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.663237 15259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:07:10.663448 15259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "b3e319f8099e415191485ba1ec096229"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "997f9911fe64e53ed2f0477187d2a396"
server_key_iv: "3f2a4a8d3fba1d3682c96f7c7397cb95"
server_key_version: "encryptionkey@0"
I20260925 11:07:10.663677 15259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:10.684397 15259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:10.689131 15259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:10.689472 15259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:10.690621 15259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:10.692073 15259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:10.692185 15259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.692340 15259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:10.692414 15259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:10.728365 15259 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:33927
I20260925 11:07:10.728406 15386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:33927 every 8 connection(s)
I20260925 11:07:10.729696 15259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:10.739825 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15259
I20260925 11:07:10.739971 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:10.740437 8916 external_mini_cluster.cc:1428] Setting key b355b33bd44ecf14f8da6d5badf889bc
Sep 25 11:07:10 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334430, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:10.753970 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:10.732515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:50739 (local address 127.8.181.62:36487)
0925 11:07:10.733028 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:10.733033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:10.734047 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 11:07:10.736280 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:10.736285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:10.736369 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:10.736471 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:10.738393 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.739272 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.741407 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.742073 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.742202 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:10.746413 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:10.746422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:10.746430 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:10.746510 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:10.750093 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:10.751302 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:10.751308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:10.751310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:10.751467 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:10.752046 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:10.752050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:10.752052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:10.752375 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:10.752503 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:10.753479 (+ 976us) server_negotiation.cc:299] Negotiation successful
0925 11:07:10.753688 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260925 11:07:10.755571 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:10.733099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:50739)
0925 11:07:10.733857 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:10.733903 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:10.735919 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:10.736865 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:10.736875 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:10.737512 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:10.738168 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:10.738189 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.739448 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.739452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:10.741224 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:10.741245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:10.742318 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:10.742323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:10.742693 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:10.743462 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:10.743494 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:10.746114 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:10.750308 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:10.750319 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:10.750340 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:10.751126 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:10.751608 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:10.751616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:10.751623 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:10.751885 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:10.752519 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:10.752530 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:10.753039 (+ 509us) client_negotiation.cc:769] Sending connection context
0925 11:07:10.753382 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 11:07:10.754012 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":287,"threads_started":1}
I20260925 11:07:10.757236 15387 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:10.757580 15387 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:10.758347 15387 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:10.760186 15064 ts_manager.cc:194] Registered new tserver with Master: b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927)
I20260925 11:07:10.761056 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:50739
WARNING: no policy specified for oryx/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
I20260925 11:07:10.799116 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:11.145668 15393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:11.145972 15393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:11.146063 15393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:11.153463 15393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:11.153606 15393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:11.153724 15393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:11.164140 15393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:11.165680 15393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:11.167070 15393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:11.183162 15398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:11.183188 15399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:11.183122 15401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:11.184005 15393 server_base.cc:1034] running on GCE node
I20260925 11:07:11.184744 15393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:11.185913 15393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:11.187201 15393 hybrid_clock.cc:648] HybridClock initialized: now 1790334431187096 us; error 67 us; skew 500 ppm
I20260925 11:07:11.187603 15393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334431, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:11.191728 15393 init.cc:377] Logged in from keytab as oryx/127.8.181.3@KRBTEST.COM (short username oryx)
I20260925 11:07:11.193486 15393 webserver.cc:460] Webserver started at http://127.8.181.3:39599/ using document root <none> and password file <none>
I20260925 11:07:11.194267 15393 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:11.194384 15393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:11.194761 15393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:11.198093 15393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "6080c5c9e7f9499481794c9046854fe1"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "510ce85d9c984c75deb69aee72d6a826"
server_key_iv: "46c9fd1345e5f526f1fd2cab14412726"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.199103 15393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "6080c5c9e7f9499481794c9046854fe1"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "510ce85d9c984c75deb69aee72d6a826"
server_key_iv: "46c9fd1345e5f526f1fd2cab14412726"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.205253 15393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:07:11.207105 15253 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
I20260925 11:07:11.209483 15408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:11.211381 15393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:11.211611 15393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "6080c5c9e7f9499481794c9046854fe1"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "510ce85d9c984c75deb69aee72d6a826"
server_key_iv: "46c9fd1345e5f526f1fd2cab14412726"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.211826 15393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:11.230398 15393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:11.235563 15393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:11.235901 15393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:11.237061 15393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:11.238679 15393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:11.238765 15393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:11.238904 15393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:11.238956 15393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:11.277463 15393 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:38491
I20260925 11:07:11.277493 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:38491 every 8 connection(s)
I20260925 11:07:11.278951 15393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:11.283561 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15393
I20260925 11:07:11.283744 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:11.284284 8916 external_mini_cluster.cc:1428] Setting key 7b26c277b6b2665ff49cb0c458fc820c
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334431, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:11.305063 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.281998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55809 (local address 127.8.181.62:36487)
0925 11:07:11.282535 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.282544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.283855 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 11:07:11.286164 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.286170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.286287 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.286418 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.288843 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.289674 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.291538 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.291970 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.292130 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.296611 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.296617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.296621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.296666 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.300892 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.301961 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.301966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.301968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.302109 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.302768 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.302772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.302774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.303084 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.303231 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.304218 (+ 987us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.304761 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":224,"threads_started":1}
I20260925 11:07:11.306069 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.282607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:55809)
0925 11:07:11.283463 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:11.283505 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:11.285846 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:11.286770 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:11.286783 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:11.287633 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:11.288501 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.288523 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.289862 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.289868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.291356 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.291371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.292237 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.292246 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.292524 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.293304 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:11.293330 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:11.296141 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:11.301131 (+ 4990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.301209 (+ 78us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.301233 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.301804 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.302245 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.302249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.302252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.302523 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.303248 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:11.303256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:11.303678 (+ 422us) client_negotiation.cc:769] Sending connection context
0925 11:07:11.303950 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 11:07:11.304459 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":382,"threads_started":1}
I20260925 11:07:11.308163 15521 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:11.308574 15521 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:11.309414 15521 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:11.311610 15064 ts_manager.cc:194] Registered new tserver with Master: 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491)
I20260925 11:07:11.312718 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:55809
I20260925 11:07:11.315222 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.61.keytab.
Entry for principal oryx/127.8.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.61.keytab.
I20260925 11:07:11.380332 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:43129
--webserver_interface=127.8.181.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.61.keytab
--principal=oryx/127.8.181.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36487,127.8.181.61:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:11.762313 15527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:11.762714 15527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:11.762836 15527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 11:07:11.764061 15387 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
W20260925 11:07:11.771160 15527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:11.771323 15527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:11.771427 15527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:11.771528 15527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:11.771605 15527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:11.782269 15527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36487,127.8.181.61:43129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.61.keytab
--principal=oryx/127.8.181.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.181.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
W20260925 11:07:11.782732 15527 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.181.62:36487,127.8.181.61:43129'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 11:07:11.783876 15527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:11.785568 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:11.798753 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:11.798808 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:11.798808 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:11.799530 15527 server_base.cc:1034] running on GCE node
I20260925 11:07:11.800379 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:11.801847 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:11.803149 15527 hybrid_clock.cc:648] HybridClock initialized: now 1790334431803066 us; error 61 us; skew 500 ppm
I20260925 11:07:11.803570 15527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334431, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:11.807049 15527 init.cc:377] Logged in from keytab as oryx/127.8.181.61@KRBTEST.COM (short username oryx)
I20260925 11:07:11.808756 15527 webserver.cc:460] Webserver started at http://127.8.181.61:36567/ using document root <none> and password file <none>
I20260925 11:07:11.809470 15527 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:11.809608 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:11.809892 15527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:11.813126 15527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/instance:
uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "3672118e9eb6205ae68645dafba44416"
server_key_iv: "69186e6590703065f0db0f0c37079859"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.813992 15527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal/instance:
uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "3672118e9eb6205ae68645dafba44416"
server_key_iv: "69186e6590703065f0db0f0c37079859"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.821221 15527 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20260925 11:07:11.825902 15542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:11.827836 15527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 11:07:11.828017 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "3672118e9eb6205ae68645dafba44416"
server_key_iv: "69186e6590703065f0db0f0c37079859"
server_key_version: "encryptionkey@0"
I20260925 11:07:11.828227 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:11.847677 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:11.852864 15527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:11.853137 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:11.873082 15527 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.61:43129
I20260925 11:07:11.873112 15593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.61:43129 every 8 connection(s)
I20260925 11:07:11.874639 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
I20260925 11:07:11.875293 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15527
I20260925 11:07:11.880439 15594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:11.887815 15594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:11.901072 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.877695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.8.181.61:43129)
0925 11:07:11.879556 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.879570 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.879615 (+ 45us) server_negotiation.cc:407] Connection header received
0925 11:07:11.881455 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.881486 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.881901 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.882324 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.883940 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.885370 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.886809 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.887567 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.887786 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.893004 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.893012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.893025 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.893070 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.896499 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.897429 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.897436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.897444 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.897656 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.898172 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.898181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.898183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.898826 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.899080 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.899566 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.899919 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1668,"thread_start_us":1412,"threads_started":1}
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334431, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.61@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:11.915503 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.890626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.8.181.62:36487)
0925 11:07:11.892120 (+ 1494us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.892129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.892152 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:07:11.892230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.892236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.892333 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.892481 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.894051 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.895073 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.896688 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.897118 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.897268 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.909373 (+ 12105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.909378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.909382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.909426 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.912339 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.913256 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.913263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.913266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.913406 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.913819 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.913824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.913827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.914160 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.914343 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.914862 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.915104 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":1239,"threads_started":1}
I20260925 11:07:11.915553 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.890524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:35196)
0925 11:07:11.891356 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:11.891415 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:11.891587 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:11.892659 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:11.892665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:11.893377 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:11.893823 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.893842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.895231 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.895236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.896509 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.896525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.897322 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.897328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.897530 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.906773 (+ 9243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:11.906796 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:11.909099 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:11.912532 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.912552 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.912560 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.913062 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.913528 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.913533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.913537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.913673 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.914319 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:11.914329 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:11.914565 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 11:07:11.914775 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 11:07:11.915145 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":327,"threads_started":1}
I20260925 11:07:11.917733 15594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:07:11 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334431, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.61@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:11.931118 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.918659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:34120)
0925 11:07:11.918864 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:11.918882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:11.919017 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:11.919335 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:11.919339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:11.919768 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:11.919897 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.919905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.920892 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.920898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.921765 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.921777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.922463 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.922502 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.922696 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.923285 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:11.923302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:11.925409 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:11.928588 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.928594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.928600 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.928969 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.929366 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.929370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.929374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.929499 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.930017 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:11.930022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:11.930163 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 11:07:11.930315 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 11:07:11.930714 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 11:07:11.931176 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.918678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.8.181.61:43129)
0925 11:07:11.918854 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.918861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.918950 (+ 89us) server_negotiation.cc:407] Connection header received
0925 11:07:11.919128 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.919132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.919225 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.919319 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.920026 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.920753 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.921895 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.922318 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.922447 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.925596 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.925601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.925604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.925631 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.928423 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.929093 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.929098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.929102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.929251 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.929609 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.929614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.929617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.929905 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.930056 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.930389 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.930848 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":45}
I20260925 11:07:11.936997 15594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Bootstrap starting.
I20260925 11:07:11.940696 15594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:11.941835 15594 log.cc:824] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:11.945166 15594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: No bootstrap required, opened a new log
I20260925 11:07:11.953419 15594 raft_consensus.cc:348] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } }
I20260925 11:07:11.953794 15594 raft_consensus.cc:374] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:11.953938 15594 raft_consensus.cc:729] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b488c6f89e4a4120aafc62c1e15b5e7d, State: Initialized, Role: FOLLOWER
I20260925 11:07:11.954784 15594 consensus_queue.cc:260] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } }
I20260925 11:07:11.955856 15601 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } } }
I20260925 11:07:11.956185 15601 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:11.956976 15594 sys_catalog.cc:564] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 11:07:11.966807 15612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:07:11.966976 15612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:07:11.969657 15527 master_runner.cc:428] Detected that this master b488c6f89e4a4120aafc62c1e15b5e7d is joining an existing cluster
I20260925 11:07:11.969791 15527 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.181.61:43129
I20260925 11:07:11.992197 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.975019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.8.181.61:43129)
0925 11:07:11.975278 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.975286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.976827 (+ 1541us) server_negotiation.cc:407] Connection header received
0925 11:07:11.976996 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.977002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.977087 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.977218 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.978660 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.979788 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.981179 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.981954 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.982177 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.984625 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.984631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.984636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.984672 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.988801 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.989559 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.989568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.989572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.989737 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.990172 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.990176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.990179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.990633 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.990819 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.991266 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.991781 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":280,"server-negotiator.queue_time_us":71}
I20260925 11:07:11.992458 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.974688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.8.181.62:36487)
0925 11:07:11.974902 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:11.974909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:11.976008 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 11:07:11.976224 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:11.976231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:11.976347 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:11.976457 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:11.978026 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.979023 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.981138 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.981628 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.981776 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.984364 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:11.984370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:11.984373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:11.984410 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:11.988651 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.989721 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.989727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.989731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.989898 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:11.990347 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:11.990352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:11.990355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:11.990784 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:11.990950 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:11.991832 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 11:07:11.992082 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 11:07:11.993300 15618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.974314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:34132)
0925 11:07:11.976705 (+ 2391us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:11.976724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:11.976854 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:11.977385 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:11.977390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:11.978118 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:11.978461 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.978507 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.979954 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.979960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.981018 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.981059 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.982184 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.982193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.982373 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.983194 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:11.983218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:11.984419 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:11.988914 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.988922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.988927 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.989415 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.989867 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.989871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.989875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.990044 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.990769 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:11.990774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:11.990946 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 11:07:11.991125 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 11:07:11.991502 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2250,"thread_start_us":1232,"threads_started":1}
I20260925 11:07:11.994225 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:11.974291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:35202)
0925 11:07:11.975851 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:11.975877 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:11.976089 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:11.976703 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:11.976708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:11.977490 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:11.977850 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.977859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.979188 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.979194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.980940 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:11.980956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:11.981855 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:11.981866 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:11.982050 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:11.983042 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:11.983073 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:11.984105 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:11.988869 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.988877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.988886 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.989552 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.990026 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:11.990032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:11.990036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:11.990212 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:11.991007 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:11.991010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:11.991248 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 11:07:11.991393 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 11:07:11.993726 (+ 2333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":44,"thread_start_us":2050,"threads_started":1}
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334429, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.62@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:12.022121 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.001379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.8.181.61:43129)
0925 11:07:12.001576 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.001582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.002185 (+ 603us) server_negotiation.cc:407] Connection header received
0925 11:07:12.002837 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.002843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.002960 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.003136 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.004396 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.005567 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.007946 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.008689 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.008858 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.013878 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.013890 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.013897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.013949 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.018220 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.019373 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.019380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.019384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.019563 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.020211 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.020217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.020220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.020604 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.020765 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.021438 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.021723 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:07:12.022328 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.001240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:34142)
0925 11:07:12.001997 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.002044 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.002491 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.003227 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.003232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.003987 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.004221 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.004231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.005749 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.005753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.007485 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.007504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.008913 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.008920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.009167 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.010080 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.010107 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.013565 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.018435 (+ 4870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.018457 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.018486 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.019188 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.019815 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.019824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.019830 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.020054 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.020782 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.020795 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.021062 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.021295 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.021760 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":360,"threads_started":1}
I20260925 11:07:12.025015 15064 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.181.61:43129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
member_type: NON_VOTER
last_known_addr {
host: "127.8.181.61"
port: 43129
}
attrs {
promote: true
}
}
dest_uuid: "b71bfda482564e31996febbc4bcd4cc4"
cas_config_opid_index: -1
I20260925 11:07:12.026273 15064 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.8.181.60@KRBTEST.COM; defaulting to no policy
I20260925 11:07:12.033332 15623 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b488c6f89e4a4120aafc62c1e15b5e7d (127.8.181.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } }
I20260925 11:07:12.035326 15569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:12.038553 15623 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.181.61:43129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.60@KRBTEST.COM" created.
I20260925 11:07:12.039407 15527 master.cc:537] Master@127.8.181.61:43129 shutting down...
I20260925 11:07:12.040548 15569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 11:07:12.039450 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } }
I20260925 11:07:12.040838 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:12.042156 15624 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
W20260925 11:07:12.046597 15049 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 11:07:12.051425 15049 consensus_peers.cc:487] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 -> Peer b488c6f89e4a4120aafc62c1e15b5e7d (127.8.181.61:43129): Couldn't send request to peer b488c6f89e4a4120aafc62c1e15b5e7d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 11:07:12.052475 15527 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:12.052940 15527 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:12.053128 15527 tablet_replica.cc:330] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: stopping tablet replica
I20260925 11:07:12.060672 15527 master.cc:552] Master@127.8.181.61:43129 shutdown complete.
I20260925 11:07:12.060781 15527 master_runner.cc:305] Clearing existing system tablet
I20260925 11:07:12.061232 15527 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:12.061326 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:12.064850 15527 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.004s
I20260925 11:07:12.066666 15629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:12.067045 15527 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:12.067390 15527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 11:07:12.067471 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "3672118e9eb6205ae68645dafba44416"
server_key_iv: "69186e6590703065f0db0f0c37079859"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.072921 15527 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.60.keytab.
Entry for principal oryx/127.8.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.60.keytab.
I20260925 11:07:12.075982 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:42183
--webserver_interface=127.8.181.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.60.keytab
--principal=oryx/127.8.181.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:07:12.080542 15527 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 11:07:12.080772 15527 log.cc:1196] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 11:07:12.081249 15527 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Deleting consensus metadata
I20260925 11:07:12.081527 15527 master_runner.cc:315] Copying system tablet from 127.8.181.62:36487
I20260925 11:07:12.088814 15527 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.181.62:36487
I20260925 11:07:12.107532 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.089773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:35212)
0925 11:07:12.090928 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.090949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.091128 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.091591 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.091597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.092153 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.092420 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.092433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.094419 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.094426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.096030 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.096049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.097358 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.097365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.097548 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.098375 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.098401 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.099546 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.104212 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.104219 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.104225 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.104787 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.105324 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.105330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.105335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.105534 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.106256 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.106262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.106454 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.106674 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.107014 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":891,"threads_started":1}
I20260925 11:07:12.107534 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.089857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.8.181.62:36487)
0925 11:07:12.090058 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.090064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.091083 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 11:07:12.091295 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.091304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.091402 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.091553 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.092612 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.094191 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.096275 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.097137 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.097329 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.099835 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.099845 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.099853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.099919 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.103986 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.104977 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.104984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.104988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.105174 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.105689 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.105695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.105698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.106095 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.106278 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.106846 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.107102 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:07:12.108775 15084 tablet_copy_service.cc:140] P b71bfda482564e31996febbc4bcd4cc4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b488c6f89e4a4120aafc62c1e15b5e7d ({username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:35212)
I20260925 11:07:12.109083 15084 tablet_copy_service.cc:161] P b71bfda482564e31996febbc4bcd4cc4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b488c6f89e4a4120aafc62c1e15b5e7d at {username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:35212: session id = b488c6f89e4a4120aafc62c1e15b5e7d-00000000000000000000000000000000
I20260925 11:07:12.114017 15084 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 11:07:12.116822 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 11:07:12.124699 15527 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: tablet copy: Starting download of 0 data blocks...
I20260925 11:07:12.125065 15527 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: tablet copy: Starting download of 1 WAL segments...
I20260925 11:07:12.128473 15527 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 11:07:12.134042 15084 tablet_copy_service.cc:342] P b71bfda482564e31996febbc4bcd4cc4: Request end of tablet copy session b488c6f89e4a4120aafc62c1e15b5e7d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:35212
I20260925 11:07:12.134352 15084 tablet_copy_service.cc:433] P b71bfda482564e31996febbc4bcd4cc4: ending tablet copy session b488c6f89e4a4120aafc62c1e15b5e7d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b488c6f89e4a4120aafc62c1e15b5e7d
I20260925 11:07:12.147369 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:12.151005 15637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:12.155489 15638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:12.162556 15640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:12.164145 15527 server_base.cc:1034] running on GCE node
I20260925 11:07:12.164783 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:12.165614 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:12.166913 15527 hybrid_clock.cc:648] HybridClock initialized: now 1790334432166830 us; error 48 us; skew 500 ppm
I20260925 11:07:12.167182 15527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:12.169616 15527 init.cc:377] Logged in from keytab as oryx/127.8.181.61@KRBTEST.COM (short username oryx)
I20260925 11:07:12.179344 15527 webserver.cc:460] Webserver started at http://127.8.181.61:43103/ using document root <none> and password file <none>
I20260925 11:07:12.179958 15527 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:12.180094 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:12.182713 15527 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 11:07:12.184593 15647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:12.185082 15527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 11:07:12.185232 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "b488c6f89e4a4120aafc62c1e15b5e7d"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "3672118e9eb6205ae68645dafba44416"
server_key_iv: "69186e6590703065f0db0f0c37079859"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.185403 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:12.196519 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:12.200096 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:12.235671 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.61:43129 every 8 connection(s)
I20260925 11:07:12.236037 15527 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.61:43129
I20260925 11:07:12.237066 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
I20260925 11:07:12.242307 15699 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:07:12.246551 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Bootstrap starting.
I20260925 11:07:12.266574 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:07:12.267170 15699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Bootstrap complete.
I20260925 11:07:12.268349 15699 raft_consensus.cc:348] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } }
I20260925 11:07:12.268658 15699 raft_consensus.cc:729] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b488c6f89e4a4120aafc62c1e15b5e7d, State: Initialized, Role: LEARNER
I20260925 11:07:12.268903 15699 consensus_queue.cc:260] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } }
I20260925 11:07:12.271083 15702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } }
I20260925 11:07:12.271528 15699 sys_catalog.cc:564] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:12.271576 15702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: LEARNER
I20260925 11:07:12.302886 15713 catalog_manager.cc:1259] Loaded cluster ID: 7075dcfbc4b14228bfe253f037ba1e62
I20260925 11:07:12.303027 15713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: loading cluster ID for follower catalog manager: success
I20260925 11:07:12.307214 15713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: acquiring CA information for follower catalog manager: success
I20260925 11:07:12.310056 15713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:07:12.316422 15521 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
W20260925 11:07:12.507483 15630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:12.507812 15630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:12.507901 15630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:12.515694 15630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:12.515825 15630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:12.515928 15630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:12.516027 15630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:12.516116 15630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:12.530810 15630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.60.keytab
--principal=oryx/127.8.181.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.181.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:12.532843 15630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:12.534732 15630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 11:07:12.540640 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.532621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.8.181.61:43129)
0925 11:07:12.534096 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.534103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.534127 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:07:12.534207 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.534211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.534307 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.534456 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:12.535050 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.536172 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.538659 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.539550 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.539661 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.539844 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.540284 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.540389 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1166,"threads_started":1}
I20260925 11:07:12.540757 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.532213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:49376)
0925 11:07:12.532743 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.532767 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.532918 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.534567 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.534574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.534608 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:12.534850 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.534864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.536353 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.536362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.538437 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.538461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.539800 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.539818 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.540023 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.540110 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.540223 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.540380 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":581,"threads_started":1}
I20260925 11:07:12.542302 15674 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 11:07:12.543179 15702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71bfda482564e31996febbc4bcd4cc4. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } }
I20260925 11:07:12.543440 15702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: LEARNER
W20260925 11:07:12.550091 15720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:12.550256 15721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:12.550128 15723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:12.551337 15630 server_base.cc:1034] running on GCE node
I20260925 11:07:12.552265 15630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:12.553857 15630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:12.555145 15630 hybrid_clock.cc:648] HybridClock initialized: now 1790334432555123 us; error 118 us; skew 500 ppm
I20260925 11:07:12.555586 15630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:12.559176 15630 init.cc:377] Logged in from keytab as oryx/127.8.181.60@KRBTEST.COM (short username oryx)
I20260925 11:07:12.561131 15630 webserver.cc:460] Webserver started at http://127.8.181.60:38811/ using document root <none> and password file <none>
I20260925 11:07:12.561960 15630 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:12.562119 15630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:12.562458 15630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:12.566154 15630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/instance:
uuid: "f660ca132d804300ba3bad58f0b1873d"
format_stamp: "Formatted at 2026-09-25 11:07:12 on dist-test-slave-6042"
server_key: "98484a46f3a1e06e44a29eb8ae01ebdf"
server_key_iv: "f9357ccd963cc66d26d5eec4eff3e5bb"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.567000 15630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal/instance:
uuid: "f660ca132d804300ba3bad58f0b1873d"
format_stamp: "Formatted at 2026-09-25 11:07:12 on dist-test-slave-6042"
server_key: "98484a46f3a1e06e44a29eb8ae01ebdf"
server_key_iv: "f9357ccd963cc66d26d5eec4eff3e5bb"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.573225 15630 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:07:12.577445 15730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:12.579375 15630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 11:07:12.579596 15630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "f660ca132d804300ba3bad58f0b1873d"
format_stamp: "Formatted at 2026-09-25 11:07:12 on dist-test-slave-6042"
server_key: "98484a46f3a1e06e44a29eb8ae01ebdf"
server_key_iv: "f9357ccd963cc66d26d5eec4eff3e5bb"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.579934 15630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:12.596436 15630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:12.601482 15630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:12.601820 15630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:12.623363 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.60:42183 every 8 connection(s)
I20260925 11:07:12.623391 15630 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.60:42183
I20260925 11:07:12.624977 15630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
I20260925 11:07:12.629796 15782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:12.633463 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15630
I20260925 11:07:12.638008 15782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.60@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:12.660374 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.642400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.8.181.62:36487)
0925 11:07:12.642995 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.643003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.643633 (+ 630us) server_negotiation.cc:407] Connection header received
0925 11:07:12.643838 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.643843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.643919 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.644017 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.645395 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.646252 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.648152 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.648606 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.648758 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.652758 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.652765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.652768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.652806 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.656341 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.657332 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.657340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.657345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.657509 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.658001 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.658038 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.658042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.658556 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.658718 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.659690 (+ 972us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.659974 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
I20260925 11:07:12.661051 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.636077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.8.181.60:42183)
0925 11:07:12.636953 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.636967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.637011 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:07:12.638836 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.638861 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.639332 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.639715 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.642005 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.644193 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.645716 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.646560 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.646759 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.651697 (+ 4938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.651711 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.651734 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.651782 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.655344 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.656537 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.656545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.656548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.656831 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.657461 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.657467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.657471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.658264 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.658653 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.659125 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.659593 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"spinlock_wait_cycles":28416,"thread_start_us":404,"threads_started":1}
I20260925 11:07:12.661114 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.642299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:38736)
0925 11:07:12.643456 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.643503 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.643696 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.644121 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.644128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.644831 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.645196 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.645207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.646412 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.646417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.647983 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.647995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.648866 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.648873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.649034 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.649708 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.649732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.652522 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.656528 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.656538 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.656551 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.657187 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.657647 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.657652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.657655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.657855 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.658811 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.658820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.659068 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.659314 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.659743 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":65,"spinlock_wait_cycles":20224,"thread_start_us":655,"threads_started":1}
I20260925 11:07:12.662887 15782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.60@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:12.677186 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.663940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.8.181.61:43129)
0925 11:07:12.664167 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.664172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.664194 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:07:12.664323 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.664331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.664427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.664511 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.665426 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.666268 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.667700 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.668127 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.668279 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.671787 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.671793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.671796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.671833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.674648 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.675403 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.675407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.675410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.675645 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.675996 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.676001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.676003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.676288 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.676401 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.676774 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.676955 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 11:07:12.677189 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.663822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:49378)
0925 11:07:12.664041 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.664063 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.664177 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.664676 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.664682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.665060 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.665274 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.665282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.666420 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.666427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.667459 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.667530 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.668292 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.668297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.668474 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.669114 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.669132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.671543 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.674809 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.674813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.674816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.675265 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.675762 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.675765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.675767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.675900 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.676415 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.676418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.676559 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.676679 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.676919 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 11:07:12.678427 15782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.60@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
I20260925 11:07:12.692502 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.679272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:55942)
0925 11:07:12.679455 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.679477 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.679590 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.679884 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.679888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.680271 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.680425 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.680445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.681702 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.681707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.682637 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.682649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.683447 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.683451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.683752 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.684443 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.684460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.687136 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.690050 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.690055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.690057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.690423 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.691032 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.691037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.691040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.691188 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.691711 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.691714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.691834 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.691946 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.692175 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 11:07:12.692647 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.679391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.8.181.60:42183)
0925 11:07:12.679551 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.679556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.679577 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:12.679698 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.679701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.679757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.679838 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.680582 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.681538 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.682806 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.683298 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.683402 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.687327 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.687332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.687336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.687366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.689881 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.690739 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.690744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.690746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.690891 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.691305 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.691309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.691311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.691596 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.691771 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.692056 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.692303 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":55}
I20260925 11:07:12.697947 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Bootstrap starting.
I20260925 11:07:12.701514 15782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:12.702821 15782 log.cc:824] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:12.706029 15782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: No bootstrap required, opened a new log
I20260925 11:07:12.714269 15782 raft_consensus.cc:348] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } }
I20260925 11:07:12.714838 15782 raft_consensus.cc:374] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:12.714995 15782 raft_consensus.cc:729] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f660ca132d804300ba3bad58f0b1873d, State: Initialized, Role: FOLLOWER
I20260925 11:07:12.715780 15782 consensus_queue.cc:260] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } }
I20260925 11:07:12.717052 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } } }
I20260925 11:07:12.717423 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:12.718253 15782 sys_catalog.cc:564] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 11:07:12.728189 15800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:07:12.728353 15800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:07:12.735908 15630 master_runner.cc:428] Detected that this master f660ca132d804300ba3bad58f0b1873d is joining an existing cluster
I20260925 11:07:12.736055 15630 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.181.60:42183
I20260925 11:07:12.759925 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.740866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.8.181.61:43129)
0925 11:07:12.741053 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.741059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.742035 (+ 976us) server_negotiation.cc:407] Connection header received
0925 11:07:12.742323 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.742331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.742417 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.742553 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.745289 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.746427 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.748532 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.749202 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.749310 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.752330 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.752337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.752343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.752388 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.756331 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.757293 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.757299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.757303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.757459 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.757967 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.757973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.757977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.758354 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.758517 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.759221 (+ 704us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.759514 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:07:12.761272 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.740908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:49388)
0925 11:07:12.741913 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.741933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.742156 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.743986 (+ 1830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.743990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.744673 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.745086 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.745106 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.746687 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.746693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.748351 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.748366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.749706 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.749713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.749895 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.750856 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.750880 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.752086 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.756503 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.756509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.756513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.757134 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.757617 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.757621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.757625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.757833 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.758614 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.758620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.758846 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.760236 (+ 1390us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.760666 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":627,"threads_started":1}
I20260925 11:07:12.769670 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15125
I20260925 11:07:12.774042 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.740842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.8.181.62:36487)
0925 11:07:12.741050 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.741058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.754823 (+ 13765us) server_negotiation.cc:407] Connection header received
0925 11:07:12.755022 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.755030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.755159 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.755260 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.757660 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.758836 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.761248 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.762022 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.762112 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.765721 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.765728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.765736 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.765779 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.769828 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.770900 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.770906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.770910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.771071 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.771626 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.771647 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.771650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.772106 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.772294 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.773227 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.773519 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:07:12.775517 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.741488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:38750)
0925 11:07:12.754673 (+ 13185us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.754697 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.754873 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.755518 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.755522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.756845 (+ 1323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.757459 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.757472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.759010 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.759019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.761040 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.761051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.762586 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.762592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.762873 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.764073 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.764103 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.765529 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.770051 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.770057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.770061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.770747 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.771230 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.771235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.771238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.771399 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.772388 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.772393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.772847 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.774259 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.774726 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12956,"mutex_wait_us":280,"spinlock_wait_cycles":106624,"thread_start_us":12513,"threads_started":1}
I20260925 11:07:12.778281 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.742522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.8.181.60:42183)
0925 11:07:12.743845 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.743851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.755474 (+ 11623us) server_negotiation.cc:407] Connection header received
0925 11:07:12.755647 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.755655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.755730 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.755815 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.757610 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.758834 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.761166 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.761973 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.762094 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.765592 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.765601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.765605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.765639 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.772219 (+ 6580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.773182 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.773187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.773191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.773332 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.773836 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.773840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.773843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.774485 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.774737 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.775417 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.777726 (+ 2309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212}
I20260925 11:07:12.779796 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.742151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:55952)
0925 11:07:12.755280 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.755299 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.755554 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.756055 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.756065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.756850 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.757459 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.757472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.759051 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.759057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.760990 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.761006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.762550 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.762558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.762868 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.764073 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.764101 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.765398 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.772355 (+ 6957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.772359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.772363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.772998 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.773525 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.773530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.773533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.773697 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.774827 (+ 1130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.774831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.775056 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.778578 (+ 3522us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.778963 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12986,"mutex_wait_us":130,"spinlock_wait_cycles":9600,"thread_start_us":12069,"threads_started":1}
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.1@KRBTEST.COM
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334429, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.62@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
I20260925 11:07:12.810075 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.784235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:55958)
0925 11:07:12.784474 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.784492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.784614 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.785176 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.785181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.785906 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.786198 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.786207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.787722 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.787730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.788909 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.788922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.790107 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.790114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.790412 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.791294 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.791322 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.795902 (+ 4580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.806370 (+ 10468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.806375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.806378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.807221 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.807712 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.807719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.807723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.807950 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.808796 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.808803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.809015 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.809207 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.809727 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 11:07:12.810083 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.770847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.8.181.1:40931)
0925 11:07:12.771658 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.771669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.771701 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:07:12.771770 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.771781 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.772097 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.775114 (+ 3017us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.775521 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.778313 (+ 2792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.781960 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.783230 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.783440 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.793796 (+ 10356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.793814 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.793829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.793883 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.805432 (+ 11549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.806926 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.806938 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.806942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.807165 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.807772 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.807777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.807781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.808296 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.808700 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.809087 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.809439 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":479,"threads_started":1}
I20260925 11:07:12.810786 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.784293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.8.181.60:42183)
0925 11:07:12.784453 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.784459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.784542 (+ 83us) server_negotiation.cc:407] Connection header received
0925 11:07:12.784784 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.784789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.785006 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.785127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.786384 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.787582 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.789070 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.789909 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.790062 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.796207 (+ 6145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.796216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.796231 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.796297 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.806153 (+ 9856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.807382 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.807386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.807388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.807559 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.808110 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.808116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.808119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.808597 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.808757 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.809700 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.809934 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 11:07:12.811954 15248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:12.812343 15064 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.181.60:42183: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f660ca132d804300ba3bad58f0b1873d"
member_type: NON_VOTER
last_known_addr {
host: "127.8.181.60"
port: 42183
}
attrs {
promote: true
}
}
dest_uuid: "b71bfda482564e31996febbc4bcd4cc4"
cas_config_opid_index: 5
I20260925 11:07:12.813719 15064 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } }
I20260925 11:07:12.816090 15623 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f660ca132d804300ba3bad58f0b1873d (127.8.181.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.819864 15757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:12.826023 15757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 11:07:12.826779 15623 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.181.60:42183
I20260925 11:07:12.827122 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.827368 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:12.827358 15674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 11:07:12.827503 15630 master.cc:537] Master@127.8.181.60:42183 shutting down...
I20260925 11:07:12.829015 15811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:12.829423 15623 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 11:07:12.845290 15674 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f660ca132d804300ba3bad58f0b1873d (127.8.181.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.850327 15702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: NON_VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: true } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.850744 15702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: LEARNER
I20260925 11:07:12.852316 15811 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: attempting to promote NON_VOTER b488c6f89e4a4120aafc62c1e15b5e7d to VOTER
I20260925 11:07:12.853188 15811 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } }
I20260925 11:07:12.861521 15674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 LEARNER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 11:07:12.862188 15811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 11:07:12.863523 15736 tablet.cc:2369] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:12.865505 15630 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:12.865978 15630 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:12.866199 15630 tablet_replica.cc:330] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: stopping tablet replica
I20260925 11:07:12.871603 15673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b488c6f89e4a4120aafc62c1e15b5e7d (127.8.181.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.873549 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.872997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:55962)
0925 11:07:12.873184 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.873339 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 11:07:12.873765 15049 consensus_peers.cc:487] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 -> Peer f660ca132d804300ba3bad58f0b1873d (127.8.181.60:42183): Couldn't send request to peer f660ca132d804300ba3bad58f0b1873d. Status: Network error: Client connection negotiation failed: client connection to 127.8.181.60:42183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 11:07:12.873245 15623 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b488c6f89e4a4120aafc62c1e15b5e7d (127.8.181.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.875936 15702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.876209 15702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:12.877489 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.877764 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:12.877887 15811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:12.878230 15811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:12.887645 15630 master.cc:552] Master@127.8.181.60:42183 shutdown complete.
I20260925 11:07:12.887760 15630 master_runner.cc:305] Clearing existing system tablet
I20260925 11:07:12.888346 15630 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:12.888484 15630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:12.891618 15630 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.001s
I20260925 11:07:12.893344 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:12.893826 15630 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:12.894081 15630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 11:07:12.894198 15630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "f660ca132d804300ba3bad58f0b1873d"
format_stamp: "Formatted at 2026-09-25 11:07:12 on dist-test-slave-6042"
server_key: "98484a46f3a1e06e44a29eb8ae01ebdf"
server_key_iv: "f9357ccd963cc66d26d5eec4eff3e5bb"
server_key_version: "encryptionkey@0"
I20260925 11:07:12.899747 15630 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:12.907212 15630 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 11:07:12.907423 15630 log.cc:1196] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 11:07:12.907902 15630 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Deleting consensus metadata
I20260925 11:07:12.908182 15630 master_runner.cc:315] Copying system tablet from 127.8.181.62:36487
I20260925 11:07:12.911247 15630 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.181.62:36487
I20260925 11:07:12.927889 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.912233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.8.181.62:36487)
0925 11:07:12.912470 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:12.912477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:12.913049 (+ 572us) server_negotiation.cc:407] Connection header received
0925 11:07:12.913266 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:12.913273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:12.913364 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:12.913480 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:12.914549 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.915650 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.917661 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.918380 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.918553 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.921000 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:12.921007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:12.921012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:12.921052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:12.924796 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.925657 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.925663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.925667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.925826 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:12.926330 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:12.926336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:12.926339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:12.926706 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:12.926848 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:12.927301 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 11:07:12.927526 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 11:07:12.927995 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:12.912222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:38760)
0925 11:07:12.912905 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:12.912927 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:12.913095 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:12.913530 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:12.913535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:12.914056 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:12.914334 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.914346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.915818 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.915824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.917481 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:12.917498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:12.918597 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:12.918604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:12.918783 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:12.919600 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:12.919627 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:12.920757 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:12.924965 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.924971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.924976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.925500 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.925949 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:12.925954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:12.925958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:12.926210 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:12.926827 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:12.926832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:12.927011 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 11:07:12.927180 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 11:07:12.927511 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":364,"threads_started":1}
I20260925 11:07:12.929121 15084 tablet_copy_service.cc:140] P b71bfda482564e31996febbc4bcd4cc4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f660ca132d804300ba3bad58f0b1873d ({username='oryx', principal='oryx/127.8.181.60@KRBTEST.COM'} at 127.0.0.1:38760)
I20260925 11:07:12.929294 15084 tablet_copy_service.cc:161] P b71bfda482564e31996febbc4bcd4cc4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f660ca132d804300ba3bad58f0b1873d at {username='oryx', principal='oryx/127.8.181.60@KRBTEST.COM'} at 127.0.0.1:38760: session id = f660ca132d804300ba3bad58f0b1873d-00000000000000000000000000000000
I20260925 11:07:12.933244 15084 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 11:07:12.935468 15630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 11:07:12.942848 15630 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: tablet copy: Starting download of 0 data blocks...
I20260925 11:07:12.943194 15630 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: tablet copy: Starting download of 1 WAL segments...
I20260925 11:07:12.946913 15630 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 11:07:12.947695 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15125
I20260925 11:07:12.952009 15084 tablet_copy_service.cc:342] P b71bfda482564e31996febbc4bcd4cc4: Request end of tablet copy session f660ca132d804300ba3bad58f0b1873d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.181.60@KRBTEST.COM'} at 127.0.0.1:38760
I20260925 11:07:12.952404 15084 tablet_copy_service.cc:433] P b71bfda482564e31996febbc4bcd4cc4: ending tablet copy session f660ca132d804300ba3bad58f0b1873d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f660ca132d804300ba3bad58f0b1873d
I20260925 11:07:12.966950 15630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:12.970111 15825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:12.978070 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:40931
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:36487
--webserver_port=36973
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:12.980835 15826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:12.988310 15828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:12.992280 15630 server_base.cc:1034] running on GCE node
I20260925 11:07:12.992842 15630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:12.993530 15630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:12.994750 15630 hybrid_clock.cc:648] HybridClock initialized: now 1790334432994696 us; error 38 us; skew 500 ppm
I20260925 11:07:12.995023 15630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:12 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334432, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:12.997759 15630 init.cc:377] Logged in from keytab as oryx/127.8.181.60@KRBTEST.COM (short username oryx)
I20260925 11:07:13.010964 15630 webserver.cc:460] Webserver started at http://127.8.181.60:37003/ using document root <none> and password file <none>
I20260925 11:07:13.011343 15630 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:13.011464 15630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:13.015067 15630 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 11:07:13.017980 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:13.018630 15630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 11:07:13.018898 15630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "f660ca132d804300ba3bad58f0b1873d"
format_stamp: "Formatted at 2026-09-25 11:07:12 on dist-test-slave-6042"
server_key: "98484a46f3a1e06e44a29eb8ae01ebdf"
server_key_iv: "f9357ccd963cc66d26d5eec4eff3e5bb"
server_key_version: "encryptionkey@0"
I20260925 11:07:13.019287 15630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:13.045925 15630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:13.051424 15630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:13.079407 15630 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.60:42183
I20260925 11:07:13.079466 15887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.60:42183 every 8 connection(s)
I20260925 11:07:13.080751 15630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
I20260925 11:07:13.088662 15888 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:07:13.094583 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Bootstrap starting.
I20260925 11:07:13.109872 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:07:13.110455 15888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: Bootstrap complete.
I20260925 11:07:13.111420 15888 raft_consensus.cc:348] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } }
I20260925 11:07:13.111706 15888 raft_consensus.cc:729] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f660ca132d804300ba3bad58f0b1873d, State: Initialized, Role: LEARNER
I20260925 11:07:13.111979 15888 consensus_queue.cc:260] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } }
I20260925 11:07:13.112850 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:13.113248 15888 sys_catalog.cc:564] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:13.113425 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: LEARNER
I20260925 11:07:13.130893 15902 catalog_manager.cc:1259] Loaded cluster ID: 7075dcfbc4b14228bfe253f037ba1e62
I20260925 11:07:13.131163 15902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: loading cluster ID for follower catalog manager: success
I20260925 11:07:13.133680 15902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: acquiring CA information for follower catalog manager: success
I20260925 11:07:13.135798 15902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 11:07:13.413290 15830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:13.413625 15830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:13.413837 15830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:13.422031 15830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:13.422164 15830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:13.422361 15830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:13.433466 15830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:40931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=36973
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:13.434978 15830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:13.436353 15830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:13.451229 15908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:13.451360 15907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:13.451519 15910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:13.452023 15830 server_base.cc:1034] running on GCE node
I20260925 11:07:13.452744 15830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:13.453868 15830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:13.455156 15830 hybrid_clock.cc:648] HybridClock initialized: now 1790334433455103 us; error 48 us; skew 500 ppm
I20260925 11:07:13.455544 15830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:13 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334433, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:13.459637 15830 init.cc:377] Logged in from keytab as oryx/127.8.181.1@KRBTEST.COM (short username oryx)
I20260925 11:07:13.461426 15830 webserver.cc:460] Webserver started at http://127.8.181.1:36973/ using document root <none> and password file <none>
I20260925 11:07:13.462216 15830 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:13.462400 15830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:13.468997 15830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:13.472370 15917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:13.474579 15830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 11:07:13.474748 15830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "c3e1d5abc8084c0ea417eae24c126818"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "d3e783599697bcc90e4aede68cbc0e89"
server_key_iv: "e8bfed269f962d6905534df1b3d68f58"
server_key_version: "encryptionkey@0"
I20260925 11:07:13.475792 15830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:13.480357 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.473473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:55970)
0925 11:07:13.474034 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:13.474051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:13.474174 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:13.475019 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:13.475024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:13.475057 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:13.475261 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.475272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.476984 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.476990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.478162 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.478178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.479476 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.479493 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.479705 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.479746 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:07:13.479864 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 11:07:13.480017 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
I20260925 11:07:13.480357 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.473603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.8.181.60:42183)
0925 11:07:13.474632 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:13.474637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:13.474674 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:07:13.474754 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:13.474763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:13.474851 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:13.474990 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:13.475411 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.476785 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.478354 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.479262 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.479430 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.479651 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:13.479889 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 11:07:13.480022 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":713,"threads_started":1}
I20260925 11:07:13.481390 15863 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 11:07:13.482121 15891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71bfda482564e31996febbc4bcd4cc4. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:13.482527 15891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: LEARNER
I20260925 11:07:13.493322 15830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:13.497967 15830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:13.498273 15830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:13.499364 15830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:13.500758 15830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:13.500842 15830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:13.500972 15830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:13.501056 15830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:13.537945 15830 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:40931
I20260925 11:07:13.537971 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:40931 every 8 connection(s)
I20260925 11:07:13.539815 15830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:13.546169 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 15830
I20260925 11:07:13.546571 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15259
Sep 25 11:07:13 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334433, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
Sep 25 11:07:13 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334433, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
Sep 25 11:07:13 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334433, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
Sep 25 11:07:13 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.2@KRBTEST.COM
I20260925 11:07:13.575011 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.543501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:49231 (local address 127.8.181.62:36487)
0925 11:07:13.543976 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:13.543982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:13.545211 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 11:07:13.548924 (+ 3713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:13.548934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:13.549060 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:13.549165 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:13.552615 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.554314 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.559528 (+ 5214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.560188 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.560293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.565376 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:13.565383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:13.565388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:13.565465 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:13.569240 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.570911 (+ 1671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.570918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.570922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.571131 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.572173 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.572180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.572184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.572597 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:13.572793 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:13.574210 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0925 11:07:13.574540 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":458,"threads_started":1}
I20260925 11:07:13.577762 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.543935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:49231)
0925 11:07:13.545027 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:13.545085 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:13.548418 (+ 3333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:13.549962 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:13.549979 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:13.551115 (+ 1136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:13.552259 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.552295 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.555357 (+ 3062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.555364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.558427 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.558453 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.560674 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.560680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.561013 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.561873 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:13.561903 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:13.565111 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:13.569871 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.569880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.569900 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.570697 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.571682 (+ 985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.571687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.571694 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.571960 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.572873 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:13.572883 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:13.573401 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 11:07:13.575283 (+ 1882us) client_negotiation.cc:240] Negotiation successful
0925 11:07:13.576070 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":524,"threads_started":1}
I20260925 11:07:13.578548 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.557614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:47673 (local address 127.8.181.61:43129)
0925 11:07:13.558219 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:13.558226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:13.558910 (+ 684us) server_negotiation.cc:407] Connection header received
0925 11:07:13.559129 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:13.559134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:13.559224 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:13.559301 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:13.560832 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.562031 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.564202 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.564848 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.564951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.571771 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:13.571778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:13.571782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:13.571825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:13.575429 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576353 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.576359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.576362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.576507 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576976 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.576981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.576984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.577330 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:13.577459 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:13.578002 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 11:07:13.578237 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":403,"threads_started":1}
I20260925 11:07:13.579293 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.558159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.8.181.1:47673)
0925 11:07:13.558825 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:13.558843 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:13.558993 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:13.559527 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:13.559531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:13.560260 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:13.560657 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.560667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.562224 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.562230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.564035 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.564049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.565949 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.565957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.566139 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.567852 (+ 1713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:13.567880 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:13.571517 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:13.575587 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.575592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.575596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.576211 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.576666 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.576678 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.576845 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.577545 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:13.577552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:13.577709 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 11:07:13.578567 (+ 858us) client_negotiation.cc:240] Negotiation successful
0925 11:07:13.578881 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":3,"thread_start_us":421,"threads_started":1}
I20260925 11:07:13.579829 16033 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:13.580195 16033 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:13.580255 16034 heartbeater.cc:347] Connected to a master server at 127.8.181.61:43129
I20260925 11:07:13.580369 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.543330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:34901 (local address 127.8.181.60:42183)
0925 11:07:13.543536 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:13.543542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:13.556446 (+ 12904us) server_negotiation.cc:407] Connection header received
0925 11:07:13.558859 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:13.558869 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:13.558964 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:13.559041 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:13.560998 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.562124 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.564427 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.565044 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.565168 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.569889 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:13.569895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:13.569899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:13.569940 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:13.574667 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576140 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.576147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.576151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.576333 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576820 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.576825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.576828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.577198 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:13.577357 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:13.579741 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0925 11:07:13.579993 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:07:13.580384 16034 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:13.581352 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.543936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.8.181.1:34901)
0925 11:07:13.556288 (+ 12352us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:13.556318 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:13.558443 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:13.559288 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:13.559292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:13.560045 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:13.560795 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.560803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.562376 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.562381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.564265 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:13.564280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.565455 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.565460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:13.565572 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.566504 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:13.566528 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:13.569682 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:13.574847 (+ 5165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.574854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.574860 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.575967 (+ 1107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.576492 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:13.576497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:13.576500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:13.576664 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:13.577446 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:13.577452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:13.577626 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 11:07:13.580537 (+ 2911us) client_negotiation.cc:240] Negotiation successful
0925 11:07:13.580872 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12118,"mutex_wait_us":538,"spinlock_wait_cycles":4992,"thread_start_us":12862,"threads_started":1}
I20260925 11:07:13.581383 16033 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:13.582537 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:13.548204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.8.181.2:33927)
0925 11:07:13.556968 (+ 8764us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:13.556976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:13.559252 (+ 2276us) server_negotiation.cc:407] Connection header received
0925 11:07:13.559445 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:13.559458 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:13.559746 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:13.559919 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:13.562263 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.563934 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.566156 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:13.567294 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:13.567424 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:13.573671 (+ 6247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:13.573692 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:13.573714 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:13.573764 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:13.578023 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.579085 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.579094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.579101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.579262 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:13.579746 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:13.579750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:13.579752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:13.580166 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:13.581766 (+ 1600us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:13.581928 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 11:07:13.582137 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8596,"thread_start_us":8424,"threads_started":1}
I20260925 11:07:13.582564 16032 heartbeater.cc:347] Connected to a master server at 127.8.181.60:42183
I20260925 11:07:13.582876 16032 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:13.582883 16034 heartbeater.cc:511] Master 127.8.181.61:43129 requested a full tablet report, sending...
I20260925 11:07:13.583365 16032 heartbeater.cc:511] Master 127.8.181.60:42183 requested a full tablet report, sending...
I20260925 11:07:13.584112 15064 ts_manager.cc:194] Re-registered known tserver with Master: c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:13.584383 15382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:13.585333 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:49231
I20260925 11:07:13.585731 15853 ts_manager.cc:194] Registered new tserver with Master: c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:13.585875 15664 ts_manager.cc:194] Registered new tserver with Master: c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:13.681432 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15259
I20260925 11:07:13.698132 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:33927
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:36487
--webserver_port=37099
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:07:14.033911 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: NON_VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: true } } }
I20260925 11:07:14.034386 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:14.035837 15815 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: attempting to promote NON_VOTER f660ca132d804300ba3bad58f0b1873d to VOTER
I20260925 11:07:14.036453 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } }
I20260925 11:07:14.038715 15863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 LEARNER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 11:07:14.038755 15673 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Refusing update from remote peer b71bfda482564e31996febbc4bcd4cc4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 11:07:14.039318 15815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 11:07:14.039768 16046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 11:07:14.042769 15815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f660ca132d804300ba3bad58f0b1873d (127.8.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.044848 15863 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f660ca132d804300ba3bad58f0b1873d (127.8.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.046605 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.047091 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:14.047365 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.047676 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:14.050009 15673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f660ca132d804300ba3bad58f0b1873d (127.8.181.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.053279 16053 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.053815 16053 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:14.059302 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.059719 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:14.060526 16056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b71bfda482564e31996febbc4bcd4cc4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:14.061017 16056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 11:07:14.063372 16044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:14.063763 16044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:14.063933 16044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:14.071815 16044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:14.071974 16044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:14.072154 16044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:14.083684 16044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:33927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=37099
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:14.085292 16044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:14.087039 16044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:14.100304 16058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:14.100282 16061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:14.100252 16059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:14.101593 16044 server_base.cc:1034] running on GCE node
I20260925 11:07:14.102267 16044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:14.103363 16044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:14.104787 16044 hybrid_clock.cc:648] HybridClock initialized: now 1790334434104752 us; error 113 us; skew 500 ppm
I20260925 11:07:14.105161 16044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:14.108786 16044 init.cc:377] Logged in from keytab as oryx/127.8.181.2@KRBTEST.COM (short username oryx)
I20260925 11:07:14.110430 16044 webserver.cc:460] Webserver started at http://127.8.181.2:37099/ using document root <none> and password file <none>
I20260925 11:07:14.111274 16044 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:14.111398 16044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:14.117396 16044 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:14.120741 16068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.122622 16044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:14.122828 16044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "b3e319f8099e415191485ba1ec096229"
format_stamp: "Formatted at 2026-09-25 11:07:10 on dist-test-slave-6042"
server_key: "997f9911fe64e53ed2f0477187d2a396"
server_key_iv: "3f2a4a8d3fba1d3682c96f7c7397cb95"
server_key_version: "encryptionkey@0"
I20260925 11:07:14.124010 16044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:14.155071 16044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:14.159775 16044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:14.160130 16044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:14.161257 16044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:14.162838 16044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:14.162954 16044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.163082 16044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:14.163179 16044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.198136 16044 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:33927
I20260925 11:07:14.198153 16180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:33927 every 8 connection(s)
I20260925 11:07:14.199648 16044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:14.204022 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16044
I20260925 11:07:14.204388 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15393
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.3@KRBTEST.COM
I20260925 11:07:14.241425 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.202954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:49189 (local address 127.8.181.62:36487)
0925 11:07:14.222675 (+ 19721us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.222682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.222706 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:07:14.222785 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.222793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.222911 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.223131 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.224217 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.225705 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.228249 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.228959 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.229172 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.233385 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.233395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.233402 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.233465 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.237690 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.238658 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.238665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.238669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.238825 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.239314 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.239321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.239327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.239703 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.239855 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.240740 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.240996 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19428,"thread_start_us":19529,"threads_started":1}
I20260925 11:07:14.242499 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.203425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:49189)
0925 11:07:14.207143 (+ 3718us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.207199 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.210262 (+ 3063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.223144 (+ 12882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.223150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.223616 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.223963 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.223976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.226394 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.226398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.228081 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.228096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.229442 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.229449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.229669 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.230371 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.230413 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.233112 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.237857 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.237863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.237868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.238533 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.238922 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.238926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.238930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.239166 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.239821 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.239830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.240275 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.240598 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.241146 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3398,"thread_start_us":23113,"threads_started":1}
I20260925 11:07:14.244170 16182 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:14.244488 16182 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.244536 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.205490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.8.181.3:38491)
0925 11:07:14.218879 (+ 13389us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.218890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.218922 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:07:14.218985 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.218993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.219312 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.219479 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.221873 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.223484 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.225429 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.226273 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.226399 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.235565 (+ 9166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.235588 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.235611 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.235673 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.239674 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.240523 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.240532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.240534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.240738 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.241554 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.241562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.241567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.242000 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.242314 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.243158 (+ 844us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.243430 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13078,"thread_start_us":12929,"threads_started":1}
I20260925 11:07:14.245486 16182 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:14.245877 15516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:14.245909 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.202545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:41461 (local address 127.8.181.60:42183)
0925 11:07:14.203291 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.203299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.215191 (+ 11892us) server_negotiation.cc:407] Connection header received
0925 11:07:14.215526 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.215535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.215641 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.215765 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.220255 (+ 4490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.221664 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.225429 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.226075 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.226172 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.232721 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.232728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.232733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.232775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.236831 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.239508 (+ 2677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.239515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.239518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.239661 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.244306 (+ 4645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.244313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.244317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.244649 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.244781 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.245315 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.245541 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":511,"threads_started":1}
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:14.247195 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.203676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.8.181.2:41461)
0925 11:07:14.215032 (+ 11356us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.215067 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.215334 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.216761 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.216781 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.218702 (+ 1921us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.219834 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.219866 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.221883 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.221889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.225264 (+ 3375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.225277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.227241 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.227251 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.227610 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.229078 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.229111 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.232461 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.237585 (+ 5124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.237596 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.237617 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.239386 (+ 1769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.242875 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.242881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.242885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.243049 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.244858 (+ 1809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.244863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.245019 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.246523 (+ 1504us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.246808 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11182,"spinlock_wait_cycles":20224,"thread_start_us":3034,"threads_started":1}
I20260925 11:07:14.247239 15064 ts_manager.cc:194] Re-registered known tserver with Master: b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927)
I20260925 11:07:14.248240 16181 heartbeater.cc:347] Connected to a master server at 127.8.181.60:42183
I20260925 11:07:14.248425 16181 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.248679 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:49189
I20260925 11:07:14.248986 16181 heartbeater.cc:511] Master 127.8.181.60:42183 requested a full tablet report, sending...
I20260925 11:07:14.250725 15853 ts_manager.cc:194] Registered new tserver with Master: b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927)
I20260925 11:07:14.255951 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.207941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:39627 (local address 127.8.181.61:43129)
0925 11:07:14.220139 (+ 12198us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.220145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.227192 (+ 7047us) server_negotiation.cc:407] Connection header received
0925 11:07:14.227431 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.227437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.227511 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.227587 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.228928 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.230040 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.231866 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.235062 (+ 3196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.235167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.249160 (+ 13993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.249168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.249173 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.249218 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.252695 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.253563 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.253569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.253574 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.253721 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.254142 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.254146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.254149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.254414 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.254674 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.255159 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.255463 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9507,"thread_start_us":9281,"threads_started":1}
I20260925 11:07:14.256088 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.209765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.8.181.2:39627)
0925 11:07:14.227088 (+ 17323us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.227108 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.227293 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.227801 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.227805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.228513 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.228774 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.228782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.230217 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.230224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.231658 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.231673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.237122 (+ 5449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.237129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.237323 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.238207 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.238227 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.248813 (+ 10586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.252847 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.252855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.252862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.253391 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.253853 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.253860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.253865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.254035 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.254627 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.254632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.254787 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.254947 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.255444 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17148,"thread_start_us":4939,"threads_started":1}
I20260925 11:07:14.257184 16183 heartbeater.cc:347] Connected to a master server at 127.8.181.61:43129
I20260925 11:07:14.257361 16183 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.257771 16183 heartbeater.cc:511] Master 127.8.181.61:43129 requested a full tablet report, sending...
I20260925 11:07:14.258932 15664 ts_manager.cc:194] Registered new tserver with Master: b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927)
I20260925 11:07:14.340596 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15393
I20260925 11:07:14.356976 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:38491
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:36487
--webserver_port=39599
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:07:14.588058 16033 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
W20260925 11:07:14.709689 16194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:14.709995 16194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:14.710098 16194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:14.717756 16194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:14.717888 16194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:14.718003 16194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:14.729063 16194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:38491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=39599
--tserver_master_addrs=127.8.181.62:36487,127.8.181.61:43129,127.8.181.60:42183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:14.730424 16194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:14.731760 16194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:14.745642 16201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:14.745604 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:14.745807 16200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:14.746529 16194 server_base.cc:1034] running on GCE node
I20260925 11:07:14.747236 16194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:14.748268 16194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:14.749519 16194 hybrid_clock.cc:648] HybridClock initialized: now 1790334434749482 us; error 70 us; skew 500 ppm
I20260925 11:07:14.749878 16194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:14.753427 16194 init.cc:377] Logged in from keytab as oryx/127.8.181.3@KRBTEST.COM (short username oryx)
I20260925 11:07:14.755121 16194 webserver.cc:460] Webserver started at http://127.8.181.3:39599/ using document root <none> and password file <none>
I20260925 11:07:14.755807 16194 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:14.755923 16194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:14.761796 16194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:14.764859 16210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.766433 16194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:07:14.766683 16194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "6080c5c9e7f9499481794c9046854fe1"
format_stamp: "Formatted at 2026-09-25 11:07:11 on dist-test-slave-6042"
server_key: "510ce85d9c984c75deb69aee72d6a826"
server_key_iv: "46c9fd1345e5f526f1fd2cab14412726"
server_key_version: "encryptionkey@0"
I20260925 11:07:14.767982 16194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:14.796530 16194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:14.801323 16194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:14.801630 16194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:14.802848 16194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:14.804498 16194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:14.804586 16194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.804744 16194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:14.804833 16194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:14.841120 16194 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:38491
I20260925 11:07:14.841192 16322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:38491 every 8 connection(s)
I20260925 11:07:14.843034 16194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:14.851568 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16194
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.60@KRBTEST.COM
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
Sep 25 11:07:14 dist-test-slave-6042 krb5kdc[15019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334434, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.61@KRBTEST.COM
I20260925 11:07:14.890445 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.846381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52023 (local address 127.8.181.62:36487)
0925 11:07:14.868297 (+ 21916us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.868303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.868325 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:07:14.868386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.868391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.868467 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.868889 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.869991 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.871219 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.874020 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.874901 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.874996 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.881720 (+ 6724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.881729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.881734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.881787 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.886096 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.887216 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.887225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.887231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.887387 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.887901 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.887906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.887910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.888299 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.888448 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.889463 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.889767 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21761,"thread_start_us":21639,"threads_started":1}
I20260925 11:07:14.891155 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.846294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:43573 (local address 127.8.181.60:42183)
0925 11:07:14.866261 (+ 19967us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.866270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.866296 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:07:14.866366 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.866372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.866517 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.866695 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.869399 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.870695 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.874038 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.874827 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.874955 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.881348 (+ 6393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.881357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.881365 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.881416 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.886013 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.888540 (+ 2527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.888545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.888548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.888686 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.889238 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.889244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.889250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.889591 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.889717 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.890227 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.890464 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19807,"thread_start_us":19703,"threads_started":1}
I20260925 11:07:14.891423 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.847133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:52023)
0925 11:07:14.865559 (+ 18426us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.865587 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.865748 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.868657 (+ 2909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.868662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.869404 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.869766 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.869782 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.871364 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.871373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.873833 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.873842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.875362 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.875371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.875816 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.876995 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.877032 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.881301 (+ 4269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.886235 (+ 4934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.886242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.886265 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.887085 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.887549 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.887554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.887563 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.887760 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.888486 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.888496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.888946 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.889314 (+ 368us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.889878 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18167,"mutex_wait_us":248,"spinlock_wait_cycles":59008,"thread_start_us":11488,"threads_started":1}
I20260925 11:07:14.892174 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.847133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.8.181.3:43573)
0925 11:07:14.860105 (+ 12972us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.860199 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.864320 (+ 4121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.867071 (+ 2751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.867090 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.868118 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.869195 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.869218 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.870969 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.870975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.873825 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.873842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.875362 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.875371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.875821 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.877011 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.877040 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.880996 (+ 3956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.886231 (+ 5235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.886242 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.886265 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.888429 (+ 2164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.888824 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.888831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.888837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.889027 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.889787 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.889795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.889946 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.891421 (+ 1475us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.891724 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11864,"mutex_wait_us":237,"spinlock_wait_cycles":23168,"thread_start_us":17820,"threads_started":1}
I20260925 11:07:14.893572 16323 heartbeater.cc:347] Connected to a master server at 127.8.181.60:42183
I20260925 11:07:14.893841 16323 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.894547 16323 heartbeater.cc:511] Master 127.8.181.60:42183 requested a full tablet report, sending...
I20260925 11:07:14.894610 16324 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36487
I20260925 11:07:14.894753 16324 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.895277 16324 heartbeater.cc:511] Master 127.8.181.62:36487 requested a full tablet report, sending...
I20260925 11:07:14.896909 15853 ts_manager.cc:194] Registered new tserver with Master: 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491)
I20260925 11:07:14.898272 16332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.865950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:40953 (local address 127.8.181.61:43129)
0925 11:07:14.866500 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.866507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.867470 (+ 963us) server_negotiation.cc:407] Connection header received
0925 11:07:14.867636 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.867643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.867756 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.867876 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.870604 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.873001 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.875205 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.876041 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.876195 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.882381 (+ 6186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.882391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.882398 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.882460 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.886727 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.887835 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.887844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.887850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.888044 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.892081 (+ 4037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.892088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.892092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.892492 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.892665 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.897585 (+ 4920us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.897850 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260925 11:07:14.898893 16333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.866523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.8.181.3:40953)
0925 11:07:14.867338 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.867357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.867490 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.868066 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.868071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.870013 (+ 1942us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:14.870365 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.870380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.873209 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.873213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.874958 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.874977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.876489 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.876496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.876702 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.877410 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:14.877445 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:14.882102 (+ 4657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:14.886917 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.886926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.886931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.887670 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.888186 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:14.888192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:14.888196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:14.888347 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:14.892727 (+ 4380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:14.892732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:14.892891 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.896380 (+ 3489us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.896722 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":508,"threads_started":1}
I20260925 11:07:14.899289 15064 ts_manager.cc:194] Re-registered known tserver with Master: 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491)
I20260925 11:07:14.900811 15064 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:52023
I20260925 11:07:14.901935 16325 heartbeater.cc:347] Connected to a master server at 127.8.181.61:43129
I20260925 11:07:14.902096 16325 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:14.902668 16325 heartbeater.cc:511] Master 127.8.181.61:43129 requested a full tablet report, sending...
I20260925 11:07:14.903914 15664 ts_manager.cc:194] Registered new tserver with Master: 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491)
I20260925 11:07:14.916544 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.874381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.8.181.62:36487)
0925 11:07:14.890770 (+ 16389us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.890779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.896281 (+ 5502us) server_negotiation.cc:407] Connection header received
0925 11:07:14.897113 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.897121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.897215 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.897309 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.901869 (+ 4560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.902970 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.906064 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.906687 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.906814 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.910033 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.910038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.910041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.910083 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.913208 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.913975 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.913981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.913985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.914168 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.914601 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.914608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.914614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.915018 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.915295 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.915721 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.916164 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16274,"thread_start_us":21171,"threads_started":1}
I20260925 11:07:14.917137 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.895786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.8.181.61:43129)
0925 11:07:14.896406 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.896412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.897141 (+ 729us) server_negotiation.cc:407] Connection header received
0925 11:07:14.897197 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.897201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.897270 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.897360 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.900257 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.901710 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.905031 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.905701 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.905827 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.910014 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.910022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.910026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.910095 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.913753 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.914691 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.914695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.914698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.914816 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.915237 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.915242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.915245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.915604 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.915822 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.916331 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.916714 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":381,"threads_started":1}
I20260925 11:07:14.920503 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.897044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.8.181.60:42183)
0925 11:07:14.897207 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.897212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.903046 (+ 5834us) server_negotiation.cc:407] Connection header received
0925 11:07:14.903309 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.903314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.903406 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.903490 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:14.905647 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.906743 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.908989 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.909679 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.909818 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.913391 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:14.913416 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:14.913421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:14.913468 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:14.917034 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.917823 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.917831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.917836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.917985 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:14.918878 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:14.918884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:14.918887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:14.919293 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:14.919580 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.919907 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.920209 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 11:07:14.921208 15064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:14.928993 15064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:14.969326 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.951066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.8.181.2:33927)
0925 11:07:14.962436 (+ 11370us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.962446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.963149 (+ 703us) server_negotiation.cc:407] Connection header received
0925 11:07:14.963390 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.963397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.963615 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.963859 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:14.964406 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.965454 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.967368 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.968289 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.968445 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.968691 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.968897 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.969055 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11205,"thread_start_us":11105,"threads_started":1}
I20260925 11:07:14.969364 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.950708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:33927 (local address 127.0.0.1:46458)
0925 11:07:14.962983 (+ 12275us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.963007 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.963197 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.963851 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.963856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.963882 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:14.964237 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.964250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.965661 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.965666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.967214 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.967228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.968498 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.968504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.968692 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.968733 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.968853 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.968988 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12147,"threads_started":1}
I20260925 11:07:14.977315 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 6591242f3cb44550b9df4eb56b500165 (DEFAULT_TABLE table=test-table [id=0d46a844a70e4426b7de97d259742006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:14.978802 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6591242f3cb44550b9df4eb56b500165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:14.978960 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.950955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40931 (local address 127.0.0.1:60998)
0925 11:07:14.969614 (+ 18659us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.969633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.969732 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.971677 (+ 1945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.971682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.971708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:14.972002 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.972015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.974519 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.974527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.976446 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.976465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.978092 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.978098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.978284 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.978327 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.978421 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.978623 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18518,"thread_start_us":11748,"threads_started":1}
I20260925 11:07:14.980422 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.951203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.8.181.1:40931)
0925 11:07:14.970992 (+ 19789us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.971002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.971069 (+ 67us) server_negotiation.cc:407] Connection header received
0925 11:07:14.971180 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.971186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.971447 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.971676 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:14.972473 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.974314 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.976605 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.977909 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.978076 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.978290 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.980042 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.980195 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19604,"thread_start_us":19470,"threads_started":1}
I20260925 11:07:14.984675 15967 tablet_service.cc:1467] Processing CreateTablet for tablet 6591242f3cb44550b9df4eb56b500165 (DEFAULT_TABLE table=test-table [id=0d46a844a70e4426b7de97d259742006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:14.986281 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.950303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.8.181.3:38491)
0925 11:07:14.975020 (+ 24717us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:14.975028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:14.975063 (+ 35us) server_negotiation.cc:407] Connection header received
0925 11:07:14.975160 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:14.975168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:14.975428 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:14.975638 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:14.978371 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.979842 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.982989 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.984792 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.985091 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.985324 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:14.985702 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 11:07:14.985879 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24526,"thread_start_us":24337,"threads_started":1}
I20260925 11:07:14.986382 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6591242f3cb44550b9df4eb56b500165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:14.986500 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:14.949857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:38491 (local address 127.0.0.1:38234)
0925 11:07:14.957140 (+ 7283us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:14.957169 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:14.957360 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:14.975673 (+ 18313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:14.975678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:14.975708 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:14.976205 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.976224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.980023 (+ 3799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.980029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.982638 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:14.982657 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:14.985202 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:14.985210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:14.985485 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:14.985521 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 11:07:14.985633 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 11:07:14.985892 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6968,"thread_start_us":6792,"threads_started":1}
I20260925 11:07:14.990942 16258 tablet_service.cc:1467] Processing CreateTablet for tablet 6591242f3cb44550b9df4eb56b500165 (DEFAULT_TABLE table=test-table [id=0d46a844a70e4426b7de97d259742006]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:14.992445 16258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6591242f3cb44550b9df4eb56b500165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:14.995378 16356 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Bootstrap starting.
I20260925 11:07:14.999823 16356 tablet_bootstrap.cc:654] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:15.001489 16356 log.cc:824] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:15.002501 16357 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Bootstrap starting.
I20260925 11:07:15.006884 16357 tablet_bootstrap.cc:654] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:15.008339 16357 log.cc:824] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:15.008414 16359 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Bootstrap starting.
I20260925 11:07:15.009684 16356 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: No bootstrap required, opened a new log
I20260925 11:07:15.010141 16356 ts_tablet_manager.cc:1397] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260925 11:07:15.011972 16357 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: No bootstrap required, opened a new log
I20260925 11:07:15.012262 16359 tablet_bootstrap.cc:654] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:15.012290 16357 ts_tablet_manager.cc:1397] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 11:07:15.013784 16359 log.cc:824] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:15.021200 16359 tablet_bootstrap.cc:492] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: No bootstrap required, opened a new log
I20260925 11:07:15.020964 16357 raft_consensus.cc:348] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.021376 16357 raft_consensus.cc:374] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:15.021471 16357 raft_consensus.cc:729] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e1d5abc8084c0ea417eae24c126818, State: Initialized, Role: FOLLOWER
I20260925 11:07:15.021529 16359 ts_tablet_manager.cc:1397] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260925 11:07:15.022166 16357 consensus_queue.cc:260] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.022248 16356 raft_consensus.cc:348] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.022699 16356 raft_consensus.cc:374] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:15.022809 16356 raft_consensus.cc:729] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e319f8099e415191485ba1ec096229, State: Initialized, Role: FOLLOWER
I20260925 11:07:15.023631 16356 consensus_queue.cc:260] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.024490 16357 ts_tablet_manager.cc:1428] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 11:07:15.032356 16356 ts_tablet_manager.cc:1428] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s
I20260925 11:07:15.032727 16182 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
I20260925 11:07:15.037645 16359 raft_consensus.cc:348] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.038012 16359 raft_consensus.cc:374] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:15.038139 16359 raft_consensus.cc:729] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6080c5c9e7f9499481794c9046854fe1, State: Initialized, Role: FOLLOWER
I20260925 11:07:15.038954 16359 consensus_queue.cc:260] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.040148 16324 heartbeater.cc:503] Master 127.8.181.62:36487 was elected leader, sending a full tablet report...
I20260925 11:07:15.041070 16359 ts_tablet_manager.cc:1428] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
W20260925 11:07:15.051039 16035 tablet.cc:2369] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:15.073201 16363 raft_consensus.cc:482] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:15.073647 16363 raft_consensus.cc:504] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.075981 16363 leader_election.cc:288] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491), c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:15.084633 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.077651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:37219 (local address 127.8.181.3:38491)
0925 11:07:15.077855 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.077861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.077881 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:07:15.077963 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.077970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.078073 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.078178 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:15.079067 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.080236 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.082593 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.083584 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.083747 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.083873 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.084218 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.084368 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:07:15.085132 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.076354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:38491 (local address 127.8.181.2:37219)
0925 11:07:15.077587 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:15.077609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:15.077772 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:15.078369 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:15.078374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:15.078412 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:15.078913 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:15.078928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.080447 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.080453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:15.082353 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:15.082370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.083829 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.083835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:15.084012 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.084047 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 11:07:15.084154 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 11:07:15.084316 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":29,"spinlock_wait_cycles":896,"thread_start_us":888,"threads_started":1}
I20260925 11:07:15.085817 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.076695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:45151 (local address 127.8.181.1:40931)
0925 11:07:15.076976 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.076986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.078335 (+ 1349us) server_negotiation.cc:407] Connection header received
0925 11:07:15.078760 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.078765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.078908 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.078979 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:15.079706 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.080673 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.083404 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.084619 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.084776 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.084892 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.085432 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.085615 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 11:07:15.086838 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.076631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40931 (local address 127.8.181.2:45151)
0925 11:07:15.078238 (+ 1607us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:15.078258 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:15.078425 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:15.079167 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:15.079170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:15.079191 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:15.079555 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:15.079566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.080823 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.080830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:15.083186 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:15.083203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.085053 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.085058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:15.085192 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.085260 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 11:07:15.086097 (+ 837us) client_negotiation.cc:240] Negotiation successful
0925 11:07:15.086385 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1497,"mutex_wait_us":1,"spinlock_wait_cycles":3328,"thread_start_us":1922,"threads_started":1}
I20260925 11:07:15.086721 16278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6591242f3cb44550b9df4eb56b500165" candidate_uuid: "b3e319f8099e415191485ba1ec096229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6080c5c9e7f9499481794c9046854fe1" is_pre_election: true
I20260925 11:07:15.087397 16278 raft_consensus.cc:2454] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e319f8099e415191485ba1ec096229 in term 0.
I20260925 11:07:15.088001 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6591242f3cb44550b9df4eb56b500165" candidate_uuid: "b3e319f8099e415191485ba1ec096229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e1d5abc8084c0ea417eae24c126818" is_pre_election: true
I20260925 11:07:15.088275 16069 leader_election.cc:302] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6080c5c9e7f9499481794c9046854fe1, b3e319f8099e415191485ba1ec096229; no voters:
I20260925 11:07:15.088507 15987 raft_consensus.cc:2454] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e319f8099e415191485ba1ec096229 in term 0.
I20260925 11:07:15.088812 16363 raft_consensus.cc:2781] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:15.088999 16363 raft_consensus.cc:482] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:15.089131 16363 raft_consensus.cc:3037] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:15.091737 16363 raft_consensus.cc:504] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.092414 16363 leader_election.cc:288] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [CANDIDATE]: Term 1 election: Requested vote from peers 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491), c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:15.093194 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6591242f3cb44550b9df4eb56b500165" candidate_uuid: "b3e319f8099e415191485ba1ec096229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e1d5abc8084c0ea417eae24c126818"
I20260925 11:07:15.093461 15987 raft_consensus.cc:3037] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:15.094372 16278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6591242f3cb44550b9df4eb56b500165" candidate_uuid: "b3e319f8099e415191485ba1ec096229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6080c5c9e7f9499481794c9046854fe1"
I20260925 11:07:15.094633 16278 raft_consensus.cc:3037] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Advancing to term 1
W20260925 11:07:15.096315 16326 tablet.cc:2369] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:15.097113 15987 raft_consensus.cc:2454] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e319f8099e415191485ba1ec096229 in term 1.
I20260925 11:07:15.097695 16072 leader_election.cc:302] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e319f8099e415191485ba1ec096229, c3e1d5abc8084c0ea417eae24c126818; no voters:
I20260925 11:07:15.098050 16363 raft_consensus.cc:2781] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:15.098222 16278 raft_consensus.cc:2454] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e319f8099e415191485ba1ec096229 in term 1.
I20260925 11:07:15.098779 16363 raft_consensus.cc:686] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 1 LEADER]: Becoming Leader. State: Replica: b3e319f8099e415191485ba1ec096229, State: Running, Role: LEADER
I20260925 11:07:15.099458 16363 consensus_queue.cc:237] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:15.105445 15064 catalog_manager.cc:5495] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e319f8099e415191485ba1ec096229 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "b3e319f8099e415191485ba1ec096229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } health_report { overall_health: UNKNOWN } } }
I20260925 11:07:15.148415 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.142312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.8.181.2:33927)
0925 11:07:15.142721 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.142727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.142755 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:07:15.142934 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.142938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.143021 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.143163 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:15.143788 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.144510 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.145919 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.146343 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.146451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.147189 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:15.147824 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:15.147949 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.148112 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.148191 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:07:15.158164 15987 raft_consensus.cc:1261] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Refusing update from remote peer b3e319f8099e415191485ba1ec096229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:15.158181 16278 raft_consensus.cc:1261] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Refusing update from remote peer b3e319f8099e415191485ba1ec096229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:15.159385 16367 consensus_queue.cc:1035] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:15.159970 16363 consensus_queue.cc:1035] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:15.179346 16371 mvcc.cc:205] Tried to move back new op lower bound from 7333209846396813312 to 7333209846171676672. Current Snapshot: MvccSnapshot[applied={T|T < 7333209846396813312}]
I20260925 11:07:15.190775 16372 mvcc.cc:205] Tried to move back new op lower bound from 7333209846396813312 to 7333209846171676672. Current Snapshot: MvccSnapshot[applied={T|T < 7333209846396813312}]
I20260925 11:07:15.197692 15064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:15.197984 15064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:15.205405 15064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71bfda482564e31996febbc4bcd4cc4: Sending DeleteTablet for 3 replicas of tablet 6591242f3cb44550b9df4eb56b500165
I20260925 11:07:15.206646 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 6591242f3cb44550b9df4eb56b500165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:15 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:46458
I20260925 11:07:15.206769 16258 tablet_service.cc:1514] Processing DeleteTablet for tablet 6591242f3cb44550b9df4eb56b500165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:15 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:38234
I20260925 11:07:15.208078 15967 tablet_service.cc:1514] Processing DeleteTablet for tablet 6591242f3cb44550b9df4eb56b500165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:15 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:60998
I20260925 11:07:15.211527 16380 tablet_replica.cc:330] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: stopping tablet replica
I20260925 11:07:15.212061 16380 raft_consensus.cc:2229] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:15.212872 16380 raft_consensus.cc:2258] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:15.216451 16380 ts_tablet_manager.cc:1905] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:15.218811 16381 tablet_replica.cc:330] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: stopping tablet replica
I20260925 11:07:15.219352 16381 raft_consensus.cc:2229] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:15.219929 16381 raft_consensus.cc:2258] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:15.223022 16381 ts_tablet_manager.cc:1905] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:15.226583 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15032
I20260925 11:07:15.227789 15094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:15.232029 16382 tablet_replica.cc:330] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: stopping tablet replica
I20260925 11:07:15.233947 16381 ts_tablet_manager.cc:1918] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:15.234148 16381 log.cc:1196] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/6591242f3cb44550b9df4eb56b500165
I20260925 11:07:15.234759 16381 ts_tablet_manager.cc:1939] T 6591242f3cb44550b9df4eb56b500165 P 6080c5c9e7f9499481794c9046854fe1: Deleting consensus metadata
I20260925 11:07:15.242458 16380 ts_tablet_manager.cc:1918] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:15.242717 16380 log.cc:1196] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/6591242f3cb44550b9df4eb56b500165
I20260925 11:07:15.243289 16380 ts_tablet_manager.cc:1939] T 6591242f3cb44550b9df4eb56b500165 P b3e319f8099e415191485ba1ec096229: Deleting consensus metadata
I20260925 11:07:15.243602 16382 raft_consensus.cc:2229] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:15.244361 16382 raft_consensus.cc:2258] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:15.247812 16382 ts_tablet_manager.cc:1905] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:15.255676 16382 ts_tablet_manager.cc:1918] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:15.255867 16382 log.cc:1196] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/6591242f3cb44550b9df4eb56b500165
I20260925 11:07:15.256402 16382 ts_tablet_manager.cc:1939] T 6591242f3cb44550b9df4eb56b500165 P c3e1d5abc8084c0ea417eae24c126818: Deleting consensus metadata
I20260925 11:07:15.263377 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.214115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.8.181.61:43129)
0925 11:07:15.214310 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.214317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.225418 (+ 11101us) server_negotiation.cc:407] Connection header received
0925 11:07:15.227796 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.227804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.227905 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.227986 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:15.244709 (+ 16723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.245817 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.249138 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.249798 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.249937 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.256402 (+ 6465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:15.256410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:15.256416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:15.256474 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:15.260200 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.261129 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.261135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.261139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.261290 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.261725 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.261731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.261734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.262018 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:15.262298 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.262791 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.263029 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 11:07:15.278450 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.225440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.8.181.60:42183)
0925 11:07:15.225687 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.225696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.263876 (+ 38180us) server_negotiation.cc:407] Connection header received
0925 11:07:15.263950 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.263955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.264039 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.264134 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:15.266071 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.266993 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.268302 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.268794 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.268945 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.272144 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:15.272150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:15.272153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:15.272190 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:15.275473 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.276225 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.276236 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.276239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.276362 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.276775 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.276782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.276785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.277083 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:15.277361 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.277832 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.278077 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 11:07:15.448912 15048 catalog_manager.cc:4841] TS 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491): tablet 6591242f3cb44550b9df4eb56b500165 (table test-table [id=0d46a844a70e4426b7de97d259742006]) successfully deleted
W20260925 11:07:15.449280 15048 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.181.3:38491, user_credentials={real_user=slave}} blocked reactor thread for 212402us
I20260925 11:07:15.449280 15051 catalog_manager.cc:4841] TS c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931): tablet 6591242f3cb44550b9df4eb56b500165 (table test-table [id=0d46a844a70e4426b7de97d259742006]) successfully deleted
I20260925 11:07:15.449827 15050 catalog_manager.cc:4841] TS b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927): tablet 6591242f3cb44550b9df4eb56b500165 (table test-table [id=0d46a844a70e4426b7de97d259742006]) successfully deleted
W20260925 11:07:15.450013 15050 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.181.2:33927, user_credentials={real_user=slave}} blocked reactor thread for 204781us
I20260925 11:07:15.450112 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15032
I20260925 11:07:15.497087 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.477837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.8.181.61:43129)
0925 11:07:15.478031 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.478037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.479169 (+ 1132us) server_negotiation.cc:407] Connection header received
0925 11:07:15.479464 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.479471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.479559 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.479755 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:15.482935 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.484175 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.486314 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.487367 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.487524 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.490817 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:15.490823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:15.490826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:15.490864 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:15.493659 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.494414 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.494420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.494424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.494647 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.495080 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.495090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.495094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.495445 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:15.495697 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.496630 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.496810 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:07:15.498749 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:15.479880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.8.181.60:42183)
0925 11:07:15.480156 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:15.480164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:15.481647 (+ 1483us) server_negotiation.cc:407] Connection header received
0925 11:07:15.481943 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:15.481948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:15.482014 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:15.482072 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:15.484453 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.485199 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.487341 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:15.488019 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:15.488152 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:15.491578 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:15.491585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:15.491590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:15.491646 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:15.495264 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.496059 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.496070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.496073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.496198 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:15.496670 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:15.496676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:15.496680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:15.497033 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:15.497557 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:15.498136 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 11:07:15.498367 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 11:07:16.188500 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.187498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:34175)
0925 11:07:16.188093 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.188250 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":329,"threads_started":1}
W20260925 11:07:16.189021 16182 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.191372 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.190541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:46281)
0925 11:07:16.191034 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.191166 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
W20260925 11:07:16.191817 16033 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.198932 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.198020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:53193)
0925 11:07:16.198560 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.198710 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":281,"threads_started":1}
W20260925 11:07:16.199460 16324 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.708060 16395 raft_consensus.cc:482] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b71bfda482564e31996febbc4bcd4cc4)
I20260925 11:07:16.708389 16395 raft_consensus.cc:504] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } }
I20260925 11:07:16.709863 16395 leader_election.cc:288] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487), f660ca132d804300ba3bad58f0b1873d (127.8.181.60:42183)
I20260925 11:07:16.711057 16398 raft_consensus.cc:482] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b71bfda482564e31996febbc4bcd4cc4)
I20260925 11:07:16.711414 16398 raft_consensus.cc:504] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } }
I20260925 11:07:16.711968 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.710026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39184)
0925 11:07:16.711485 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.711717 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1344,"mutex_wait_us":67,"thread_start_us":382,"threads_started":1}
W20260925 11:07:16.712215 15650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.713222 16398 leader_election.cc:288] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487), b488c6f89e4a4120aafc62c1e15b5e7d (127.8.181.61:43129)
I20260925 11:07:16.714709 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.713738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39212)
0925 11:07:16.714277 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.714496 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
W20260925 11:07:16.715029 15650 leader_election.cc:334] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487): Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.716214 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.713224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39196)
0925 11:07:16.715677 (+ 2453us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.715890 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2319,"thread_start_us":2125,"threads_started":1}
W20260925 11:07:16.716557 15839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.718796 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.718253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39216)
0925 11:07:16.718425 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.718629 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 11:07:16.719053 15839 leader_election.cc:334] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487): Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.721423 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.710026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:42183 (local address 127.0.0.1:56006)
0925 11:07:16.710770 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.710795 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:16.710945 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:16.712107 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:16.712112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:16.712138 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:16.712561 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.712573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.716757 (+ 4184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.716762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.719072 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.719082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.720560 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.720567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.720751 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.720794 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 11:07:16.720908 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 11:07:16.721076 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":1183,"threads_started":1}
I20260925 11:07:16.721467 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.711065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.8.181.60:42183)
0925 11:07:16.711811 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:16.711815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:16.711832 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:16.711898 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:16.711902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:16.711955 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:16.712086 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:16.712862 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.716624 (+ 3762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.719264 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.720350 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.720507 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.720586 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:16.720946 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 11:07:16.721131 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":647,"thread_start_us":495,"threads_started":1}
I20260925 11:07:16.722168 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f660ca132d804300ba3bad58f0b1873d" is_pre_election: true
I20260925 11:07:16.722599 15863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b488c6f89e4a4120aafc62c1e15b5e7d in term 1.
I20260925 11:07:16.723201 15649 leader_election.cc:302] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b488c6f89e4a4120aafc62c1e15b5e7d, f660ca132d804300ba3bad58f0b1873d; no voters: b71bfda482564e31996febbc4bcd4cc4
I20260925 11:07:16.723594 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.713401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:43129 (local address 127.0.0.1:49422)
0925 11:07:16.716405 (+ 3004us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.716422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:16.716577 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:16.717058 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:16.717065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:16.717093 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:16.717555 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.717567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.719219 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.719227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.720934 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.720948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.722701 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.722709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.722911 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.722945 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:07:16.723063 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 11:07:16.723216 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2922,"mutex_wait_us":1,"thread_start_us":3329,"threads_started":1}
I20260925 11:07:16.723649 16395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 11:07:16.723767 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.713487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.8.181.61:43129)
0925 11:07:16.715539 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:16.715545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:16.716675 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 11:07:16.716749 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:16.716754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:16.716905 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:16.717020 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:16.717785 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.719058 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.721115 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.722473 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.722619 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.722696 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:16.723120 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 11:07:16.723243 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1924,"thread_start_us":1652,"threads_started":1}
I20260925 11:07:16.723793 16395 raft_consensus.cc:482] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader b71bfda482564e31996febbc4bcd4cc4)
I20260925 11:07:16.723891 16395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:07:16.724370 15673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f660ca132d804300ba3bad58f0b1873d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" is_pre_election: true
I20260925 11:07:16.726060 16395 raft_consensus.cc:504] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } }
I20260925 11:07:16.726519 15673 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f660ca132d804300ba3bad58f0b1873d in current term 2: Already voted for candidate b488c6f89e4a4120aafc62c1e15b5e7d in this term.
I20260925 11:07:16.726812 16395 leader_election.cc:288] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 election: Requested vote from peers b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487), f660ca132d804300ba3bad58f0b1873d (127.8.181.60:42183)
I20260925 11:07:16.727360 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.726937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39222)
0925 11:07:16.727123 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.727216 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 11:07:16.727560 15863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f660ca132d804300ba3bad58f0b1873d"
I20260925 11:07:16.727808 15863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:07:16.728202 15838 leader_election.cc:302] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f660ca132d804300ba3bad58f0b1873d; no voters: b488c6f89e4a4120aafc62c1e15b5e7d, b71bfda482564e31996febbc4bcd4cc4
I20260925 11:07:16.728464 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.728056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39236)
0925 11:07:16.728224 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.728321 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 11:07:16.728672 15650 leader_election.cc:334] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487): Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
I20260925 11:07:16.730257 15863 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b488c6f89e4a4120aafc62c1e15b5e7d in term 2.
I20260925 11:07:16.730511 16398 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:07:16.730788 15649 leader_election.cc:302] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b488c6f89e4a4120aafc62c1e15b5e7d, f660ca132d804300ba3bad58f0b1873d; no voters: b71bfda482564e31996febbc4bcd4cc4
I20260925 11:07:16.731087 16395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 11:07:16.731600 16395 raft_consensus.cc:686] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [term 2 LEADER]: Becoming Leader. State: Replica: b488c6f89e4a4120aafc62c1e15b5e7d, State: Running, Role: LEADER
I20260925 11:07:16.732010 16395 consensus_queue.cc:237] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } }
I20260925 11:07:16.733952 16405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b488c6f89e4a4120aafc62c1e15b5e7d. Latest consensus state: current_term: 2 leader_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:16.734345 16405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:16.735021 16407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:16.738778 16407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:16.739708 16407 catalog_manager.cc:1259] Loaded cluster ID: 7075dcfbc4b14228bfe253f037ba1e62
I20260925 11:07:16.739781 16407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:16.741201 16407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:16.742282 16407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Loaded TSK: 0
I20260925 11:07:16.743180 16407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:07:16.796561 15664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:16.804425 15664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:16.809015 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.808600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39260)
0925 11:07:16.808774 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.808870 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 11:07:16.809064 15863 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [term 2 FOLLOWER]: Refusing update from remote peer b488c6f89e4a4120aafc62c1e15b5e7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 11:07:16.810098 16405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 11:07:16.810395 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.809825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39276)
0925 11:07:16.810136 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.810230 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260925 11:07:16.810606 15650 consensus_peers.cc:487] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d -> Peer b71bfda482564e31996febbc4bcd4cc4 (127.8.181.62:36487): Couldn't send request to peer b71bfda482564e31996febbc4bcd4cc4. Status: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 11:07:16.813241 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b488c6f89e4a4120aafc62c1e15b5e7d. Latest consensus state: current_term: 2 leader_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:16.813541 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:16.815267 16405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:16.815593 16405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:16.816144 16398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b71bfda482564e31996febbc4bcd4cc4" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36487 } } peers { permanent_uuid: "b488c6f89e4a4120aafc62c1e15b5e7d" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 43129 } attrs { promote: false } } peers { permanent_uuid: "f660ca132d804300ba3bad58f0b1873d" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 42183 } attrs { promote: false } } }
I20260925 11:07:16.816421 16398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f660ca132d804300ba3bad58f0b1873d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:07:16.835937 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.825022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:38491 (local address 127.0.0.1:38244)
0925 11:07:16.825242 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.825271 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:16.825417 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:16.826013 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:16.826018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:16.826044 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:16.826272 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.826284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.827855 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.827860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.829301 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.829316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.834743 (+ 5427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.834753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.835128 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.835190 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 11:07:16.835287 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 11:07:16.835496 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 11:07:16.837066 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.825086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.8.181.3:38491)
0925 11:07:16.825665 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:16.825670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:16.825687 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:16.825747 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:16.825750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:16.825847 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:16.826417 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:16.826639 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.827709 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.829462 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.830531 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.836255 (+ 5724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.836369 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:16.836438 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 11:07:16.836653 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":295,"threads_started":1}
I20260925 11:07:16.837997 16258 tablet_service.cc:1467] Processing CreateTablet for tablet 808805743bd04b75a858f8c07cbc1dfc (DEFAULT_TABLE table=test-table [id=09141aada5364ce8bdcd7dcf1278973f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:16.838445 16258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808805743bd04b75a858f8c07cbc1dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:16.847811 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.830247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40931 (local address 127.0.0.1:32780)
0925 11:07:16.833965 (+ 3718us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.833992 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:16.834164 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:16.837919 (+ 3755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:16.837927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:16.837967 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:16.838426 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.838440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.839954 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.839959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.844751 (+ 4792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.844774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.846818 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.846826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.847038 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.847076 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 11:07:16.847170 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 11:07:16.847333 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3541,"mutex_wait_us":32,"thread_start_us":3309,"threads_started":1}
I20260925 11:07:16.848652 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.830775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.8.181.1:40931)
0925 11:07:16.837570 (+ 6795us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:16.837577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:16.837597 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:07:16.837670 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:16.837675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:16.837746 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:16.837847 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:16.838730 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.839806 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.845031 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.846286 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.846436 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.846640 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:16.848251 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0925 11:07:16.848413 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6654,"thread_start_us":13914,"threads_started":1}
I20260925 11:07:16.848661 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.827001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:33927 (local address 127.0.0.1:46472)
0925 11:07:16.827358 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.827376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:16.827487 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:16.838998 (+ 11511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:16.839003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:16.839031 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:16.839265 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.839276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.840466 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.840471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.844839 (+ 4368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:16.844863 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.847413 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.847428 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:16.847731 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.847772 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:07:16.847849 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 11:07:16.848318 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":57}
I20260925 11:07:16.849479 15967 tablet_service.cc:1467] Processing CreateTablet for tablet 808805743bd04b75a858f8c07cbc1dfc (DEFAULT_TABLE table=test-table [id=09141aada5364ce8bdcd7dcf1278973f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:16.849943 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.827393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.8.181.2:33927)
0925 11:07:16.838707 (+ 11314us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:16.838712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:16.838727 (+ 15us) server_negotiation.cc:407] Connection header received
0925 11:07:16.838792 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:16.838795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:16.838853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:16.838982 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:16.839426 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.840326 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.845156 (+ 4830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:16.847151 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:16.849062 (+ 1911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:16.849236 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:16.849308 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 11:07:16.849519 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11181,"thread_start_us":11751,"threads_started":1}
I20260925 11:07:16.849972 15967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808805743bd04b75a858f8c07cbc1dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:16.850689 16418 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Bootstrap starting.
I20260925 11:07:16.850849 16116 tablet_service.cc:1467] Processing CreateTablet for tablet 808805743bd04b75a858f8c07cbc1dfc (DEFAULT_TABLE table=test-table [id=09141aada5364ce8bdcd7dcf1278973f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:16.851317 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 808805743bd04b75a858f8c07cbc1dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:16.852281 16418 tablet_bootstrap.cc:654] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:16.854290 16418 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: No bootstrap required, opened a new log
I20260925 11:07:16.854547 16418 ts_tablet_manager.cc:1397] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:16.855207 16418 raft_consensus.cc:348] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.855435 16418 raft_consensus.cc:374] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:16.855533 16418 raft_consensus.cc:729] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6080c5c9e7f9499481794c9046854fe1, State: Initialized, Role: FOLLOWER
I20260925 11:07:16.855782 16418 consensus_queue.cc:260] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.856874 16418 ts_tablet_manager.cc:1428] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:07:16.858083 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.856950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:38431)
0925 11:07:16.857785 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.857887 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":537,"threads_started":1}
I20260925 11:07:16.859263 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.858723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:50947)
0925 11:07:16.859067 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.859144 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260925 11:07:16.864506 16423 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Bootstrap starting.
I20260925 11:07:16.864830 16421 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Bootstrap starting.
I20260925 11:07:16.866161 16423 tablet_bootstrap.cc:654] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:16.866686 16421 tablet_bootstrap.cc:654] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:16.868412 16423 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: No bootstrap required, opened a new log
I20260925 11:07:16.868516 16421 tablet_bootstrap.cc:492] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: No bootstrap required, opened a new log
I20260925 11:07:16.868706 16423 ts_tablet_manager.cc:1397] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 11:07:16.868721 16421 ts_tablet_manager.cc:1397] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 11:07:16.869379 16423 raft_consensus.cc:348] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.869380 16421 raft_consensus.cc:348] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.869572 16421 raft_consensus.cc:374] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:16.869609 16423 raft_consensus.cc:374] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:16.869675 16421 raft_consensus.cc:729] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3e1d5abc8084c0ea417eae24c126818, State: Initialized, Role: FOLLOWER
I20260925 11:07:16.869726 16423 raft_consensus.cc:729] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e319f8099e415191485ba1ec096229, State: Initialized, Role: FOLLOWER
I20260925 11:07:16.869927 16421 consensus_queue.cc:260] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.869956 16423 consensus_queue.cc:260] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:16.870862 16423 ts_tablet_manager.cc:1428] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 11:07:16.875758 16183 heartbeater.cc:503] Master 127.8.181.61:43129 was elected leader, sending a full tablet report...
I20260925 11:07:16.876211 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.871439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:37477)
0925 11:07:16.875459 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.875680 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3891,"thread_start_us":3743,"threads_started":1}
I20260925 11:07:16.877238 16421 ts_tablet_manager.cc:1428] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20260925 11:07:16.878979 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.876869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:40271)
0925 11:07:16.878701 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.878802 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1707,"threads_started":1}
I20260925 11:07:16.882792 16034 heartbeater.cc:503] Master 127.8.181.61:43129 was elected leader, sending a full tablet report...
I20260925 11:07:16.883983 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.877868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:56747)
0925 11:07:16.883400 (+ 5532us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.883612 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5327,"thread_start_us":5019,"threads_started":1}
I20260925 11:07:16.885475 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:16.885038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:55267)
0925 11:07:16.885283 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:16.885358 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260925 11:07:16.957382 16184 tablet.cc:2369] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:17.066576 16420 raft_consensus.cc:482] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:17.066852 16420 raft_consensus.cc:504] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:17.068428 16420 leader_election.cc:288] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927), c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:17.076815 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.068779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:47265 (local address 127.8.181.2:33927)
0925 11:07:17.068955 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.068963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.069069 (+ 106us) server_negotiation.cc:407] Connection header received
0925 11:07:17.069337 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.069343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.069444 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.069530 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:17.070072 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.071190 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.074305 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.075689 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.075842 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.075943 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.076396 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.076535 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:07:17.076833 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.068860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57703 (local address 127.8.181.1:40931)
0925 11:07:17.069117 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.069126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.069617 (+ 491us) server_negotiation.cc:407] Connection header received
0925 11:07:17.069778 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.069782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.069837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.069906 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:17.070359 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.071437 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.074301 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.075690 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.075842 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.075945 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.076460 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.076588 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:07:17.076841 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.068659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:33927 (local address 127.8.181.3:47265)
0925 11:07:17.068923 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.068942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:17.069135 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:17.069593 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:17.069597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:17.069642 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:17.069915 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:17.069928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.071397 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.071404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:17.074088 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:17.074096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.075929 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.075938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:17.076153 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.076208 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 11:07:17.076328 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 11:07:17.076494 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":109,"spinlock_wait_cycles":7936}
I20260925 11:07:17.077721 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.068848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40931 (local address 127.8.181.3:57703)
0925 11:07:17.069477 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.069491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:17.069651 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:17.069953 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:17.069960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:17.069991 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:17.070219 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:17.070225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.071551 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.071568 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:17.074067 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:17.074092 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.076090 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.076101 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:17.076237 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.076261 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 11:07:17.077108 (+ 847us) client_negotiation.cc:240] Negotiation successful
0925 11:07:17.077229 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":82,"spinlock_wait_cycles":11648,"thread_start_us":388,"threads_started":1}
I20260925 11:07:17.078238 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808805743bd04b75a858f8c07cbc1dfc" candidate_uuid: "6080c5c9e7f9499481794c9046854fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e319f8099e415191485ba1ec096229" is_pre_election: true
I20260925 11:07:17.078544 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808805743bd04b75a858f8c07cbc1dfc" candidate_uuid: "6080c5c9e7f9499481794c9046854fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e1d5abc8084c0ea417eae24c126818" is_pre_election: true
I20260925 11:07:17.078774 15987 raft_consensus.cc:2454] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6080c5c9e7f9499481794c9046854fe1 in term 0.
I20260925 11:07:17.078783 16136 raft_consensus.cc:2454] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6080c5c9e7f9499481794c9046854fe1 in term 0.
I20260925 11:07:17.079416 16214 leader_election.cc:302] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6080c5c9e7f9499481794c9046854fe1, b3e319f8099e415191485ba1ec096229, c3e1d5abc8084c0ea417eae24c126818; no voters:
I20260925 11:07:17.079919 16420 raft_consensus.cc:2781] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:17.080067 16420 raft_consensus.cc:482] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:17.080163 16420 raft_consensus.cc:3037] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:17.083932 16420 raft_consensus.cc:504] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:17.084856 16420 leader_election.cc:288] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [CANDIDATE]: Term 1 election: Requested vote from peers b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927), c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931)
I20260925 11:07:17.085536 15987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808805743bd04b75a858f8c07cbc1dfc" candidate_uuid: "6080c5c9e7f9499481794c9046854fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3e1d5abc8084c0ea417eae24c126818"
I20260925 11:07:17.085722 15987 raft_consensus.cc:3037] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:17.088231 15987 raft_consensus.cc:2454] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6080c5c9e7f9499481794c9046854fe1 in term 1.
I20260925 11:07:17.088748 16136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "808805743bd04b75a858f8c07cbc1dfc" candidate_uuid: "6080c5c9e7f9499481794c9046854fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e319f8099e415191485ba1ec096229"
I20260925 11:07:17.088817 16214 leader_election.cc:302] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6080c5c9e7f9499481794c9046854fe1, c3e1d5abc8084c0ea417eae24c126818; no voters:
I20260925 11:07:17.088984 16136 raft_consensus.cc:3037] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:17.089249 16420 raft_consensus.cc:2781] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:17.089781 16420 raft_consensus.cc:686] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 6080c5c9e7f9499481794c9046854fe1, State: Running, Role: LEADER
I20260925 11:07:17.090255 16420 consensus_queue.cc:237] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } }
I20260925 11:07:17.090441 16325 heartbeater.cc:503] Master 127.8.181.61:43129 was elected leader, sending a full tablet report...
I20260925 11:07:17.091518 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.090876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:41141)
0925 11:07:17.091196 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.091346 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 11:07:17.092340 16136 raft_consensus.cc:2454] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6080c5c9e7f9499481794c9046854fe1 in term 1.
I20260925 11:07:17.095754 15663 catalog_manager.cc:5495] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3). New cstate: current_term: 1 leader_uuid: "6080c5c9e7f9499481794c9046854fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6080c5c9e7f9499481794c9046854fe1" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 } health_report { overall_health: UNKNOWN } } }
I20260925 11:07:17.168320 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.163023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.8.181.3:38491)
0925 11:07:17.163221 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.163225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.163244 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:07:17.163439 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.163442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.163523 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.163668 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:17.163999 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.164748 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.166012 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.166417 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.166607 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.167124 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:17.167777 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:17.167894 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.168039 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.168111 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:07:17.172742 15987 raft_consensus.cc:1261] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Refusing update from remote peer 6080c5c9e7f9499481794c9046854fe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:17.173139 16136 raft_consensus.cc:1261] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Refusing update from remote peer 6080c5c9e7f9499481794c9046854fe1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:17.173575 16432 consensus_queue.cc:1035] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3e1d5abc8084c0ea417eae24c126818" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:17.174018 16420 consensus_queue.cc:1035] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e319f8099e415191485ba1ec096229" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:17.176846 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.176311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:37177)
0925 11:07:17.176486 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.176612 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 11:07:17.179041 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.178099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:51311)
0925 11:07:17.178360 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.178673 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 11:07:17.186178 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.183767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.2:43713)
0925 11:07:17.185747 (+ 1980us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.185888 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1810}
I20260925 11:07:17.187567 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.186976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.1:53473)
0925 11:07:17.187193 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.187318 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 11:07:17.190706 16437 mvcc.cc:205] Tried to move back new op lower bound from 7333209854651998208 to 7333209854327865344. Current Snapshot: MvccSnapshot[applied={T|T < 7333209854651998208}]
I20260925 11:07:17.194067 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.193540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:47641)
0925 11:07:17.193736 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.193854 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 11:07:17.196888 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.196362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.8.181.3:60071)
0925 11:07:17.196589 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.196718 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 11:07:17.206171 15663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:17.206514 15663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:17.213377 15663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b488c6f89e4a4120aafc62c1e15b5e7d: Sending DeleteTablet for 3 replicas of tablet 808805743bd04b75a858f8c07cbc1dfc
I20260925 11:07:17.214633 16258 tablet_service.cc:1514] Processing DeleteTablet for tablet 808805743bd04b75a858f8c07cbc1dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:17 UTC) from {username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:38244
I20260925 11:07:17.214879 15967 tablet_service.cc:1514] Processing DeleteTablet for tablet 808805743bd04b75a858f8c07cbc1dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:17 UTC) from {username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:32780
I20260925 11:07:17.215124 16116 tablet_service.cc:1514] Processing DeleteTablet for tablet 808805743bd04b75a858f8c07cbc1dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:17 UTC) from {username='oryx', principal='oryx/127.8.181.61@KRBTEST.COM'} at 127.0.0.1:46472
I20260925 11:07:17.217128 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15830
I20260925 11:07:17.227035 16446 tablet_replica.cc:330] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: stopping tablet replica
I20260925 11:07:17.227411 16446 raft_consensus.cc:2229] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:17.227874 16446 raft_consensus.cc:2258] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:17.229396 16446 ts_tablet_manager.cc:1905] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:17.243103 16445 tablet_replica.cc:330] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: stopping tablet replica
I20260925 11:07:17.243357 16445 raft_consensus.cc:2229] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:17.243702 16445 raft_consensus.cc:2258] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:17.244457 16444 tablet_replica.cc:330] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: stopping tablet replica
I20260925 11:07:17.244685 16444 raft_consensus.cc:2229] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:17.245235 16444 raft_consensus.cc:2258] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:17.246636 16444 ts_tablet_manager.cc:1905] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:17.251772 16445 ts_tablet_manager.cc:1905] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:17.254132 16444 ts_tablet_manager.cc:1918] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:17.254325 16444 log.cc:1196] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/808805743bd04b75a858f8c07cbc1dfc
I20260925 11:07:17.254871 16444 ts_tablet_manager.cc:1939] T 808805743bd04b75a858f8c07cbc1dfc P 6080c5c9e7f9499481794c9046854fe1: Deleting consensus metadata
I20260925 11:07:17.256237 15648 catalog_manager.cc:4841] TS 6080c5c9e7f9499481794c9046854fe1 (127.8.181.3:38491): tablet 808805743bd04b75a858f8c07cbc1dfc (table test-table [id=09141aada5364ce8bdcd7dcf1278973f]) successfully deleted
I20260925 11:07:17.261348 16445 ts_tablet_manager.cc:1918] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:17.263886 16446 ts_tablet_manager.cc:1918] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:17.264153 16446 log.cc:1196] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/808805743bd04b75a858f8c07cbc1dfc
I20260925 11:07:17.264683 16446 ts_tablet_manager.cc:1939] T 808805743bd04b75a858f8c07cbc1dfc P b3e319f8099e415191485ba1ec096229: Deleting consensus metadata
I20260925 11:07:17.265812 16445 log.cc:1196] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/808805743bd04b75a858f8c07cbc1dfc
I20260925 11:07:17.266013 15650 catalog_manager.cc:4841] TS b3e319f8099e415191485ba1ec096229 (127.8.181.2:33927): tablet 808805743bd04b75a858f8c07cbc1dfc (table test-table [id=09141aada5364ce8bdcd7dcf1278973f]) successfully deleted
I20260925 11:07:17.266335 16445 ts_tablet_manager.cc:1939] T 808805743bd04b75a858f8c07cbc1dfc P c3e1d5abc8084c0ea417eae24c126818: Deleting consensus metadata
I20260925 11:07:17.267599 15651 catalog_manager.cc:4841] TS c3e1d5abc8084c0ea417eae24c126818 (127.8.181.1:40931): tablet 808805743bd04b75a858f8c07cbc1dfc (table test-table [id=09141aada5364ce8bdcd7dcf1278973f]) successfully deleted
I20260925 11:07:17.269481 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.218021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.8.181.1:40931)
0925 11:07:17.218217 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.218225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.243614 (+ 25389us) server_negotiation.cc:407] Connection header received
0925 11:07:17.243863 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.243868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.243950 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.244054 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:17.247524 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.248722 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.250798 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.251474 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.251581 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.262388 (+ 10807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:17.262409 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:17.262430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:17.262535 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:17.266104 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.267148 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.267156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.267159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.267307 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.267825 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.267830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.267833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.268196 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:17.268516 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.268874 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.269130 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 11:07:17.270093 16027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:17.326576 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.326009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39278)
0925 11:07:17.326221 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.326360 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 11:07:17.328289 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.327797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36487 (local address 127.0.0.1:39280)
0925 11:07:17.328006 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:17.328103 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 11:07:17.369310 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15830
I20260925 11:07:17.388003 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16044
I20260925 11:07:17.402618 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.388854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.8.181.2:33927)
0925 11:07:17.389022 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.389029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.389104 (+ 75us) server_negotiation.cc:407] Connection header received
0925 11:07:17.389394 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.389400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.389489 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.389597 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:17.391023 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.391802 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.393048 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.393487 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.393629 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.396509 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:17.396532 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:17.396549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:17.396588 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:17.399641 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.400417 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.400427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.400431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.400564 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.400960 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.400967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.400970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.401343 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:17.401645 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.401947 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.402155 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:07:17.403313 16176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:17.525197 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16044
I20260925 11:07:17.547348 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16194
I20260925 11:07:17.564266 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:17.548213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.8.181.3:38491)
0925 11:07:17.548389 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:17.548396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:17.548479 (+ 83us) server_negotiation.cc:407] Connection header received
0925 11:07:17.548809 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:17.548815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:17.548885 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:17.549026 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:17.550452 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.551350 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.552626 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:17.553085 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:17.553245 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:17.556597 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:17.556611 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:17.556628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:17.556674 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:17.560634 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.561666 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.561675 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.561678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.561869 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:17.562355 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:17.562359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:17.562361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:17.562925 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:17.563294 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:17.563609 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 11:07:17.563850 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:07:17.564981 16318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:17.702363 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16194
I20260925 11:07:17.725073 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15527
I20260925 11:07:17.726313 15694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:17.852895 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15527
I20260925 11:07:17.875159 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 15630
I20260925 11:07:17.876336 15883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:17.993677 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 15630
2026-09-25T11:07:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10377 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:18 dist-test-slave-6042 krb5kdc[16457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:18 dist-test-slave-6042 krb5kdc[16457](info): set up 2 sockets
Sep 25 11:07:18 dist-test-slave-6042 krb5kdc[16457](info): commencing operation
krb5kdc: starting...
W20260925 11:07:19.143952 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:19 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab.
Entry for principal oryx/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab.
I20260925 11:07:19.391885 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:46409
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab
--principal=oryx/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:19.745363 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:19.745764 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:19.745921 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:19.753621 16470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:19.753810 16470 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 11:07:19.753887 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:19.753979 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:19.754076 16470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:19.754164 16470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:19.765573 16470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:46409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.62.keytab
--principal=oryx/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:19.767225 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:19.768762 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:19.781941 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:19.782155 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:19.782176 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:19.783631 16470 server_base.cc:1034] running on GCE node
I20260925 11:07:19.784477 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:19.786052 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:19.787308 16470 hybrid_clock.cc:648] HybridClock initialized: now 1790334439787218 us; error 57 us; skew 500 ppm
I20260925 11:07:19.787743 16470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:19 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334439, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:19.791241 16470 init.cc:377] Logged in from keytab as oryx/127.8.181.62@KRBTEST.COM (short username oryx)
I20260925 11:07:19.792868 16470 webserver.cc:460] Webserver started at http://127.8.181.62:39795/ using document root <none> and password file <none>
I20260925 11:07:19.793591 16470 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:19.793684 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:19.793965 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:19.797277 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "5d1d368d267b4286a3c10d5d81cc4530"
format_stamp: "Formatted at 2026-09-25 11:07:19 on dist-test-slave-6042"
server_key: "bd1101bd0975ce4621563e849de8c544"
server_key_iv: "8d51afdcfe37c747fcc4798d550a58e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:19.798120 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "5d1d368d267b4286a3c10d5d81cc4530"
format_stamp: "Formatted at 2026-09-25 11:07:19 on dist-test-slave-6042"
server_key: "bd1101bd0975ce4621563e849de8c544"
server_key_iv: "8d51afdcfe37c747fcc4798d550a58e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:19.804054 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260925 11:07:19.808240 16485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:19.809823 16470 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:07:19.810058 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "5d1d368d267b4286a3c10d5d81cc4530"
format_stamp: "Formatted at 2026-09-25 11:07:19 on dist-test-slave-6042"
server_key: "bd1101bd0975ce4621563e849de8c544"
server_key_iv: "8d51afdcfe37c747fcc4798d550a58e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:19.810360 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:19.835397 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:19.840669 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:19.841006 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:19.866598 16470 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:46409
I20260925 11:07:19.866623 16546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:46409 every 8 connection(s)
I20260925 11:07:19.868256 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:19.872661 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:19.877744 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16470
I20260925 11:07:19.877978 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:19.878727 8916 external_mini_cluster.cc:1428] Setting key 973b2b97235fe46c0b7c14aeb7c2ef6e
I20260925 11:07:19.885475 16547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Bootstrap starting.
I20260925 11:07:19.890444 16547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:19.891870 16547 log.cc:824] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:19.895212 16547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: No bootstrap required, opened a new log
Sep 25 11:07:19 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:19.904525 16547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } }
I20260925 11:07:19.904989 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:19.880966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.8.181.62:46409)
0925 11:07:19.881887 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:19.881900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:19.881950 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:07:19.883894 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:19.883922 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:19.884367 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:19.884935 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:19.887270 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:19.888932 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:19.891126 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:19.891927 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:19.892249 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:19.896791 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:19.896800 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:19.896816 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:19.896857 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:19.900420 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:19.901238 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:19.901246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:19.901254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:19.901447 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:19.901885 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:19.901890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:19.901892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:19.902562 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:19.902912 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:19.903431 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 11:07:19.903899 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":449,"threads_started":1}
I20260925 11:07:19.905215 16547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:19.905354 16547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1d368d267b4286a3c10d5d81cc4530, State: Initialized, Role: FOLLOWER
I20260925 11:07:19.906234 16547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } }
I20260925 11:07:19.906576 16547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:19.906735 16547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:19.906957 16547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:19.909646 16547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } }
I20260925 11:07:19.910218 16547 leader_election.cc:302] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d1d368d267b4286a3c10d5d81cc4530; no voters:
I20260925 11:07:19.910910 16547 leader_election.cc:288] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:19.911154 16552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:19.911969 16552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [term 1 LEADER]: Becoming Leader. State: Replica: 5d1d368d267b4286a3c10d5d81cc4530, State: Running, Role: LEADER
I20260925 11:07:19.913363 16552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } }
I20260925 11:07:19.913743 16547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:19.916498 16553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d1d368d267b4286a3c10d5d81cc4530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } } }
I20260925 11:07:19.916944 16553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:19.916476 16554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d1d368d267b4286a3c10d5d81cc4530. Latest consensus state: current_term: 1 leader_uuid: "5d1d368d267b4286a3c10d5d81cc4530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d1d368d267b4286a3c10d5d81cc4530" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 46409 } } }
I20260925 11:07:19.917150 16554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:19.925582 16558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:19.931901 16558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:19.941354 16558 catalog_manager.cc:1347] Generated new cluster ID: a2564693acfb4c6b93a47fe657b5cde8
I20260925 11:07:19.941532 16558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:19.948997 16558 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:19.951026 16558 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:19.959285 16558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Generated new TSK 0
I20260925 11:07:19.960485 16558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
I20260925 11:07:20.028404 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:46409
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:20.388834 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:20.389230 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:20.389416 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:20.397907 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:20.398113 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:20.398286 16573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:07:20.398437 16573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:20.408906 16573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:20.410399 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:20.411983 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:20.428072 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:20.428099 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:20.428182 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:20.429028 16573 server_base.cc:1034] running on GCE node
I20260925 11:07:20.429694 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:20.430790 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:20.432106 16573 hybrid_clock.cc:648] HybridClock initialized: now 1790334440432021 us; error 53 us; skew 500 ppm
I20260925 11:07:20.432412 16573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:20 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334440, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:20.436007 16573 init.cc:377] Logged in from keytab as oryx/127.8.181.1@KRBTEST.COM (short username oryx)
I20260925 11:07:20.437731 16573 webserver.cc:460] Webserver started at http://127.8.181.1:42925/ using document root <none> and password file <none>
I20260925 11:07:20.438462 16573 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:20.438637 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:20.438967 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:20.442070 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "d7d390a712504015a95551210af4874f"
format_stamp: "Formatted at 2026-09-25 11:07:20 on dist-test-slave-6042"
server_key: "4b15508f4c4dee4335c66df9022ff41e"
server_key_iv: "feb342ad8086edee5ee02ca6045b99e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:20.442979 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "d7d390a712504015a95551210af4874f"
format_stamp: "Formatted at 2026-09-25 11:07:20 on dist-test-slave-6042"
server_key: "4b15508f4c4dee4335c66df9022ff41e"
server_key_iv: "feb342ad8086edee5ee02ca6045b99e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:20.448711 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:07:20.452965 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:20.454916 16573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 11:07:20.455157 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "d7d390a712504015a95551210af4874f"
format_stamp: "Formatted at 2026-09-25 11:07:20 on dist-test-slave-6042"
server_key: "4b15508f4c4dee4335c66df9022ff41e"
server_key_iv: "feb342ad8086edee5ee02ca6045b99e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:20.455394 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:20.472640 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:20.478174 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:20.478674 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:20.481696 16573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:20.481803 16573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:20.481968 16573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:20.482030 16573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:07:20 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334440, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:20.511132 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:20.482624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:43955 (local address 127.8.181.62:46409)
0925 11:07:20.483154 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:20.483160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:20.484501 (+ 1341us) server_negotiation.cc:407] Connection header received
0925 11:07:20.486720 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:20.486727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:20.486831 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:20.486962 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:20.490879 (+ 3917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.491635 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.495563 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.496258 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.496449 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:20.502392 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:20.502399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:20.502402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:20.502444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:20.505910 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:20.507064 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:20.507074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:20.507079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:20.507277 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:20.508001 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:20.508004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:20.508007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:20.508434 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:20.508596 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:20.510087 (+ 1491us) server_negotiation.cc:299] Negotiation successful
0925 11:07:20.510529 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260925 11:07:20.511718 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:20.483301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.1:43955)
0925 11:07:20.484274 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:20.484324 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:20.486315 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:20.487431 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:20.487453 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:20.488856 (+ 1403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:20.490323 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:20.490368 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.491815 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.491823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:20.495271 (+ 3448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:20.495297 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.496625 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.496636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:20.497112 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:20.498218 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:20.498249 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:20.502018 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:20.506111 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:20.506122 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:20.506155 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:20.506827 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:20.507409 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:20.507414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:20.507422 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:20.507819 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:20.508689 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:20.508696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:20.509287 (+ 591us) client_negotiation.cc:769] Sending connection context
0925 11:07:20.509777 (+ 490us) client_negotiation.cc:240] Negotiation successful
0925 11:07:20.510262 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":419,"threads_started":1}
I20260925 11:07:20.543027 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:20.522549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:57869 (local address 127.8.181.62:46409)
0925 11:07:20.522775 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:20.522783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:20.523640 (+ 857us) server_negotiation.cc:407] Connection header received
0925 11:07:20.524039 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:20.524046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:20.524230 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:20.524415 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:20.527409 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.528877 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.532123 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.533062 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.533200 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:20.536629 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:20.536635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:20.536639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:20.536675 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:20.540049 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:20.540898 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:20.540902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:20.540905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:20.541055 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:20.541449 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:20.541459 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:20.541461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:20.541754 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:20.541929 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:20.542362 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 11:07:20.542652 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:07:20.543731 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:20.522559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.1:57869)
0925 11:07:20.523403 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:20.523434 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:20.523799 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:20.524786 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:20.524828 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:20.526186 (+ 1358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:20.527112 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:20.527136 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.529159 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.529169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:20.531841 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:20.531873 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:20.533663 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:20.533670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:20.533943 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:20.534944 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:20.534976 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:20.536333 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:20.540227 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:20.540234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:20.540238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:20.540737 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:20.541179 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:20.541185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:20.541188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:20.541325 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:20.541879 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:20.541884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:20.542085 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 11:07:20.542244 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 11:07:20.542637 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":493,"threads_started":1}
I20260925 11:07:20.554049 16573 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:40487
I20260925 11:07:20.554075 16708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:40487 every 8 connection(s)
I20260925 11:07:20.555476 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:20.555917 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16573
I20260925 11:07:20.556131 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:20.556775 8916 external_mini_cluster.cc:1428] Setting key 613f7aa56667c4691fec47d32805de34
I20260925 11:07:20.558005 16709 heartbeater.cc:347] Connected to a master server at 127.8.181.62:46409
I20260925 11:07:20.558265 16709 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:20.559031 16709 heartbeater.cc:511] Master 127.8.181.62:46409 requested a full tablet report, sending...
I20260925 11:07:20.561334 16502 ts_manager.cc:194] Registered new tserver with Master: d7d390a712504015a95551210af4874f (127.8.181.1:40487)
I20260925 11:07:20.562975 16502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:43955
WARNING: no policy specified for oryx/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
I20260925 11:07:20.611109 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:46409
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:20.960649 16713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:20.960973 16713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:20.961143 16713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:20.968842 16713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:20.968986 16713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:20.969204 16713 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:07:20.969390 16713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:20.979897 16713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:20.981351 16713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:20.982932 16713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:20.997735 16721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:20.997682 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:20.998203 16718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:20.999433 16713 server_base.cc:1034] running on GCE node
I20260925 11:07:21.000167 16713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:21.001169 16713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:21.002413 16713 hybrid_clock.cc:648] HybridClock initialized: now 1790334441002329 us; error 56 us; skew 500 ppm
I20260925 11:07:21.002878 16713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:21.006582 16713 init.cc:377] Logged in from keytab as oryx/127.8.181.2@KRBTEST.COM (short username oryx)
I20260925 11:07:21.008246 16713 webserver.cc:460] Webserver started at http://127.8.181.2:40711/ using document root <none> and password file <none>
I20260925 11:07:21.009040 16713 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:21.009189 16713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:21.009550 16713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:21.012717 16713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "1ee7e1f7d54d93f135bd3df29a8734a1"
server_key_iv: "656af645fc37e8dca573efc146ac0313"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.013684 16713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "1ee7e1f7d54d93f135bd3df29a8734a1"
server_key_iv: "656af645fc37e8dca573efc146ac0313"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.019887 16713 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260925 11:07:21.023872 16728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:21.025525 16713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 11:07:21.025743 16713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "1ee7e1f7d54d93f135bd3df29a8734a1"
server_key_iv: "656af645fc37e8dca573efc146ac0313"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.026001 16713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:21.046567 16713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:21.051558 16713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:21.051892 16713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:21.055158 16713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:21.055274 16713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:21.055433 16713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:21.055567 16713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:21.084357 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.056517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.2:42785)
0925 11:07:21.057347 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.057393 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.059633 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.060507 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.060524 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.061413 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:21.062382 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.062405 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.064530 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.064539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.067044 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.067112 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.068476 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.068482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.068871 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.069778 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:21.069818 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:21.073274 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:21.078236 (+ 4962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.078251 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.078283 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.079076 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.079728 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.079732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.079735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.080080 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.080889 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:21.080898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:21.081528 (+ 630us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.081930 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.082650 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":349,"threads_started":1}
I20260925 11:07:21.085476 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.055848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42785 (local address 127.8.181.62:46409)
0925 11:07:21.056533 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.056537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.057523 (+ 986us) server_negotiation.cc:407] Connection header received
0925 11:07:21.059966 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.059973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.060074 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.060223 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.062701 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.064241 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.067299 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.067942 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.068134 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.073657 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:21.073667 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:21.073673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:21.073724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:21.078037 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.079303 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.079313 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.079320 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.079543 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.080297 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.080305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.080311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.080728 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:21.080912 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.084999 (+ 4087us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.085217 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":444,"threads_started":1}
I20260925 11:07:21.100423 16713 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:46847
I20260925 11:07:21.100464 16848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:46847 every 8 connection(s)
I20260925 11:07:21.102253 16713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:21.105072 16849 heartbeater.cc:347] Connected to a master server at 127.8.181.62:46409
I20260925 11:07:21.105404 16849 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:21.106364 16849 heartbeater.cc:511] Master 127.8.181.62:46409 requested a full tablet report, sending...
I20260925 11:07:21.106712 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16713
I20260925 11:07:21.106947 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:21.107467 8916 external_mini_cluster.cc:1428] Setting key 34cdcbddff67b9db1f9717d8b0ad1e8b
I20260925 11:07:21.108385 16502 ts_manager.cc:194] Registered new tserver with Master: bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847)
I20260925 11:07:21.109695 16502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:42785
I20260925 11:07:21.112521 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.093482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:54225 (local address 127.8.181.62:46409)
0925 11:07:21.093710 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.093726 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.095417 (+ 1691us) server_negotiation.cc:407] Connection header received
0925 11:07:21.095709 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.095716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.095829 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.095953 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.098176 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.099335 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.101669 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.102224 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.102341 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.105484 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:21.105492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:21.105498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:21.105546 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:21.109500 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.110415 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.110421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.110425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.110623 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.111022 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.111027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.111031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.111367 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:21.111532 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.111932 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.112168 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 11:07:21.112661 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.093118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.2:54225)
0925 11:07:21.095241 (+ 2123us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.095269 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.095490 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.096380 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.096387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.097346 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:21.097940 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.097955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.099560 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.099567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.101368 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.101384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.102460 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.102487 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.102683 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.103582 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:21.103615 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:21.105267 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:21.109695 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.109701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.109705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.110268 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.110750 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.110755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.110759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.110894 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.111499 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:21.111505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:21.111686 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.111839 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.112189 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1893,"thread_start_us":1606,"threads_started":1}
WARNING: no policy specified for oryx/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
I20260925 11:07:21.163120 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:46409
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:21.506686 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:21.506973 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:21.507073 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:21.515172 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:21.515331 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:21.515487 16853 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 11:07:21.515628 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:21.526521 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.181.62:46409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:21.528007 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:21.529599 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:21.546859 16859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:21.546921 16858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:21.547118 16861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:21.548174 16853 server_base.cc:1034] running on GCE node
I20260925 11:07:21.548846 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:21.549954 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:21.551215 16853 hybrid_clock.cc:648] HybridClock initialized: now 1790334441551121 us; error 65 us; skew 500 ppm
I20260925 11:07:21.551821 16853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:21.556356 16853 init.cc:377] Logged in from keytab as oryx/127.8.181.3@KRBTEST.COM (short username oryx)
I20260925 11:07:21.558355 16853 webserver.cc:460] Webserver started at http://127.8.181.3:40227/ using document root <none> and password file <none>
I20260925 11:07:21.559264 16853 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:21.559430 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:21.559798 16853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:21.563282 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "f2988a0a465645819fddcf22cfc095ad"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "67ca607356d27bbb30bf6da148ae0ed5"
server_key_iv: "d3c0b6da637762a7d37ff19d115620f6"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.564220 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "f2988a0a465645819fddcf22cfc095ad"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "67ca607356d27bbb30bf6da148ae0ed5"
server_key_iv: "d3c0b6da637762a7d37ff19d115620f6"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.566008 16709 heartbeater.cc:503] Master 127.8.181.62:46409 was elected leader, sending a full tablet report...
I20260925 11:07:21.571226 16853 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 11:07:21.575525 16868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:21.577318 16853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:21.577543 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "f2988a0a465645819fddcf22cfc095ad"
format_stamp: "Formatted at 2026-09-25 11:07:21 on dist-test-slave-6042"
server_key: "67ca607356d27bbb30bf6da148ae0ed5"
server_key_iv: "d3c0b6da637762a7d37ff19d115620f6"
server_key_version: "encryptionkey@0"
I20260925 11:07:21.577785 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:21.598030 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:21.603229 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:21.603585 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:21.606711 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:21.606806 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:21.606988 16853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:21.607064 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:21.641006 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.607428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:44457 (local address 127.8.181.62:46409)
0925 11:07:21.607638 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.607646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.609262 (+ 1616us) server_negotiation.cc:407] Connection header received
0925 11:07:21.612621 (+ 3359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.612631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.612858 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.612991 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.616374 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.617746 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.621737 (+ 3991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.623001 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.623252 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.631145 (+ 7893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:21.631152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:21.631156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:21.631209 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:21.635539 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.637266 (+ 1727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.637272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.637275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.637442 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.638295 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.638304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.638307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.638817 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:21.639009 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.640138 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.640479 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:07:21.643042 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.608032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.3:44457)
0925 11:07:21.609023 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.609094 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.612121 (+ 3027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.613486 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.613519 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.614550 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:21.615951 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.615994 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.618046 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.618056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.621378 (+ 3322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.621407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.623762 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.623779 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.624213 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.625609 (+ 1396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:21.625672 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:21.630718 (+ 5046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:21.635844 (+ 5126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.635858 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.635884 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.637014 (+ 1130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.637604 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.637613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.637619 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.638127 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.638984 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:21.638994 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:21.639540 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.639953 (+ 413us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.640816 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":365,"threads_started":1}
I20260925 11:07:21.658320 16853 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:34663
I20260925 11:07:21.658354 16987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:34663 every 8 connection(s)
I20260925 11:07:21.660064 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:21.662524 16988 heartbeater.cc:347] Connected to a master server at 127.8.181.62:46409
I20260925 11:07:21.662890 16988 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:21.663650 16988 heartbeater.cc:511] Master 127.8.181.62:46409 requested a full tablet report, sending...
I20260925 11:07:21.665194 16502 ts_manager.cc:194] Registered new tserver with Master: f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663)
I20260925 11:07:21.666069 16502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:44457
I20260925 11:07:21.667461 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.650767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:40577 (local address 127.8.181.62:46409)
0925 11:07:21.650998 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.651007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.651957 (+ 950us) server_negotiation.cc:407] Connection header received
0925 11:07:21.652231 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.652239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.652340 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.652478 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.653609 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.655008 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.657715 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.658254 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.658434 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.660818 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:21.660824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:21.660827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:21.660863 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:21.664479 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.665213 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.665218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.665222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.665455 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.665856 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.665862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.665866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.666161 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:21.666373 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.666856 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.667099 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:07:21.667625 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.650811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.8.181.3:40577)
0925 11:07:21.651779 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.651814 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.652030 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.652548 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.652556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.653097 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:21.653427 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.653438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.655274 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.655283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.657468 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.657505 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.658549 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.658557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.658742 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.659450 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:21.659467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:21.660585 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:21.664647 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.664653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.664658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.665072 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.665579 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:21.665585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:21.665588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:21.665737 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:21.666333 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:21.666339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:21.666599 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.666761 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.667109 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":507,"threads_started":1}
I20260925 11:07:21.667646 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 16853
I20260925 11:07:21.667799 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:21.668285 8916 external_mini_cluster.cc:1428] Setting key 4de04a597cf851911a95478b628424ff
I20260925 11:07:21.671725 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:21.730067 16737 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:21.717461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.8.181.62:46409)
0925 11:07:21.717764 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.717769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.718330 (+ 561us) server_negotiation.cc:407] Connection header received
0925 11:07:21.718653 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.718658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.718731 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.718824 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.720528 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.721304 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.722778 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.723242 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.723354 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.729462 (+ 6108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58326: BlockingRecv error: recv got EOF from 127.0.0.1:58326 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 25 11:07:21 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:21.749702 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.733684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.8.181.62:46409)
0925 11:07:21.733961 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.733967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.734296 (+ 329us) server_negotiation.cc:407] Connection header received
0925 11:07:21.734438 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.734443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.734609 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.734705 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:21.736071 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.737031 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.738879 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.739354 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.739482 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.743903 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:21.743908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:21.743912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:21.743957 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:21.746990 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.747816 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.747820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.747823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.747974 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:21.748370 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:21.748373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:21.748375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:21.748705 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:21.748908 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.749224 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.749402 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 11:07:21.754770 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:21.762243 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:21.789836 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.777543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.8.181.3:34663)
0925 11:07:21.779544 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.779557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.779625 (+ 68us) server_negotiation.cc:407] Connection header received
0925 11:07:21.779762 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.779774 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.780123 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.781954 (+ 1831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:21.782289 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.784077 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.786516 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.788508 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.788672 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.788931 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.789292 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.789473 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1744,"thread_start_us":2391,"threads_started":1}
I20260925 11:07:21.789883 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.776566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:34663 (local address 127.0.0.1:50710)
0925 11:07:21.777813 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.777867 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.778129 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.780965 (+ 2836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.780974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.781024 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:21.781587 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.781626 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.784227 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.784233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.786277 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.786292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.788705 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.788712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.788938 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.789003 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.789239 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.789481 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":21,"thread_start_us":5138,"threads_started":1}
I20260925 11:07:21.791471 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.779743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.8.181.1:40487)
0925 11:07:21.781092 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.781102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.782202 (+ 1100us) server_negotiation.cc:407] Connection header received
0925 11:07:21.782395 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.782401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.782660 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.782860 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:21.784286 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.785685 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.788367 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.789817 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.789952 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.790148 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.790993 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.791173 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1193,"thread_start_us":1057,"threads_started":1}
I20260925 11:07:21.791528 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.779088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40487 (local address 127.0.0.1:39758)
0925 11:07:21.782023 (+ 2935us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.782039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.782212 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.783083 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.783088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.783121 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:21.784082 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.784094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.785836 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.785841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.788191 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.788206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.790367 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.790372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.790675 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.790815 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.790912 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.791093 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2822,"mutex_wait_us":160,"thread_start_us":2208,"threads_started":1}
I20260925 11:07:21.792672 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.777150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.8.181.2:46847)
0925 11:07:21.780786 (+ 3636us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.780797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.781981 (+ 1184us) server_negotiation.cc:407] Connection header received
0925 11:07:21.782113 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.782121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.782433 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.782673 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:21.783780 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.785288 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.788330 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.789817 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.789956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.790150 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.792154 (+ 2004us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.792340 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3459,"thread_start_us":3318,"threads_started":1}
I20260925 11:07:21.793249 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.776980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46847 (local address 127.0.0.1:51796)
0925 11:07:21.781653 (+ 4673us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.781675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.781821 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.782917 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.782922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.782952 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:21.783587 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.783605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.785454 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.785459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.788109 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.788122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.790355 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.790362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.790674 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.790813 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.790911 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.792207 (+ 1296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4487,"mutex_wait_us":262,"thread_start_us":845,"threads_started":1}
I20260925 11:07:21.794569 16918 tablet_service.cc:1467] Processing CreateTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f (DEFAULT_TABLE table=test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:21.795104 16639 tablet_service.cc:1467] Processing CreateTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f (DEFAULT_TABLE table=test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:21.796026 16918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a9b0461eb6458ab0d880042a91bd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:21.796497 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a9b0461eb6458ab0d880042a91bd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:21.797164 16779 tablet_service.cc:1467] Processing CreateTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f (DEFAULT_TABLE table=test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:21.799809 16779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a9b0461eb6458ab0d880042a91bd9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:21.808229 17012 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Bootstrap starting.
I20260925 11:07:21.812167 17012 tablet_bootstrap.cc:654] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:21.813771 17012 log.cc:824] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:21.814877 17013 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Bootstrap starting.
I20260925 11:07:21.817698 17015 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Bootstrap starting.
I20260925 11:07:21.819762 17013 tablet_bootstrap.cc:654] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:21.821215 17015 tablet_bootstrap.cc:654] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:21.821264 17013 log.cc:824] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:21.821818 17012 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: No bootstrap required, opened a new log
I20260925 11:07:21.822122 17012 ts_tablet_manager.cc:1397] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260925 11:07:21.822525 17015 log.cc:824] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:21.828886 17013 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: No bootstrap required, opened a new log
I20260925 11:07:21.829150 17015 tablet_bootstrap.cc:492] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: No bootstrap required, opened a new log
I20260925 11:07:21.829339 17013 ts_tablet_manager.cc:1397] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260925 11:07:21.829430 17015 ts_tablet_manager.cc:1397] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260925 11:07:21.834374 17012 raft_consensus.cc:348] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.834942 17012 raft_consensus.cc:374] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:21.835044 17012 raft_consensus.cc:729] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Initialized, Role: FOLLOWER
I20260925 11:07:21.835759 17012 consensus_queue.cc:260] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.839046 17013 raft_consensus.cc:348] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.839479 16988 heartbeater.cc:503] Master 127.8.181.62:46409 was elected leader, sending a full tablet report...
I20260925 11:07:21.839555 17013 raft_consensus.cc:374] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:21.839669 17013 raft_consensus.cc:729] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d390a712504015a95551210af4874f, State: Initialized, Role: FOLLOWER
I20260925 11:07:21.840297 17012 ts_tablet_manager.cc:1428] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Time spent starting tablet: real 0.018s user 0.011s sys 0.009s
I20260925 11:07:21.840356 17013 consensus_queue.cc:260] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.842888 17013 ts_tablet_manager.cc:1428] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 11:07:21.843132 17015 raft_consensus.cc:348] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.843694 17015 raft_consensus.cc:374] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:21.843791 17015 raft_consensus.cc:729] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5bd4c8b0d469b93805e9b8bfa7735, State: Initialized, Role: FOLLOWER
I20260925 11:07:21.844573 17015 consensus_queue.cc:260] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.849005 16849 heartbeater.cc:503] Master 127.8.181.62:46409 was elected leader, sending a full tablet report...
I20260925 11:07:21.849638 17015 ts_tablet_manager.cc:1428] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
W20260925 11:07:21.855162 16850 tablet.cc:2369] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:21.877153 17018 raft_consensus.cc:482] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:21.877666 17018 raft_consensus.cc:504] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.879593 17018 leader_election.cc:288] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), d7d390a712504015a95551210af4874f (127.8.181.1:40487)
I20260925 11:07:21.886593 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.880149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:48817 (local address 127.8.181.2:46847)
0925 11:07:21.880357 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.880362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.880386 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:07:21.880464 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.880469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.880555 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.880660 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:21.881315 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.882397 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.884836 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.885615 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.885786 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.885895 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.886180 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.886309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:07:21.886590 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.879882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46847 (local address 127.8.181.3:48817)
0925 11:07:21.880133 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.880156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.880342 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.880859 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.880865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.880891 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:21.881146 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.881156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.882555 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.882561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.884654 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.884672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.885795 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.885800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.886008 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.886032 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.886146 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.886283 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 11:07:21.887820 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.880573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:37929 (local address 127.8.181.1:40487)
0925 11:07:21.880779 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.880785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.881134 (+ 349us) server_negotiation.cc:407] Connection header received
0925 11:07:21.881301 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.881306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.881379 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.881464 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:21.882088 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.883211 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.885542 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.886452 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.886695 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.886824 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.887436 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.887587 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:07:21.887735 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a9b0461eb6458ab0d880042a91bd9f" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" is_pre_election: true
I20260925 11:07:21.888036 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.880307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40487 (local address 127.8.181.3:37929)
0925 11:07:21.880978 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:21.880997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:21.881136 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:21.881517 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:21.881521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:21.881548 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:21.881881 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.881892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.883378 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.883386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.885352 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:21.885366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.886976 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.886983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:21.887241 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.887265 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 11:07:21.887407 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 11:07:21.887574 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":1,"thread_start_us":334,"threads_started":1}
I20260925 11:07:21.888355 16799 raft_consensus.cc:2454] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:21.889186 16871 leader_election.cc:302] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb5bd4c8b0d469b93805e9b8bfa7735, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:21.889194 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a9b0461eb6458ab0d880042a91bd9f" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f" is_pre_election: true
I20260925 11:07:21.889739 17018 raft_consensus.cc:2781] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:21.889724 16661 raft_consensus.cc:2454] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:21.889901 17018 raft_consensus.cc:482] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:21.890025 17018 raft_consensus.cc:3037] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:21.892383 17018 raft_consensus.cc:504] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.892995 17018 leader_election.cc:288] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Requested vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), d7d390a712504015a95551210af4874f (127.8.181.1:40487)
I20260925 11:07:21.893606 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a9b0461eb6458ab0d880042a91bd9f" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
I20260925 11:07:21.893796 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a9b0461eb6458ab0d880042a91bd9f" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f"
I20260925 11:07:21.893884 16799 raft_consensus.cc:3037] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:21.894057 16661 raft_consensus.cc:3037] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:21.896355 16661 raft_consensus.cc:2454] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:21.896469 16799 raft_consensus.cc:2454] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:21.896867 16871 leader_election.cc:302] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d390a712504015a95551210af4874f, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:21.897207 17018 raft_consensus.cc:2781] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:21.897675 17018 raft_consensus.cc:686] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Becoming Leader. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Running, Role: LEADER
I20260925 11:07:21.898182 17018 consensus_queue.cc:237] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } }
I20260925 11:07:21.903494 16502 catalog_manager.cc:5495] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f2988a0a465645819fddcf22cfc095ad (127.8.181.3). New cstate: current_term: 1 leader_uuid: "f2988a0a465645819fddcf22cfc095ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } health_report { overall_health: UNKNOWN } } }
W20260925 11:07:21.912025 16989 tablet.cc:2369] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:21.923099 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:21.916764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.8.181.3:34663)
0925 11:07:21.916960 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:21.916964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:21.917080 (+ 116us) server_negotiation.cc:407] Connection header received
0925 11:07:21.917283 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:21.917287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:21.917374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:21.917515 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:21.917937 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.918763 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.920302 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:21.920732 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:21.920847 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:21.921718 (+ 871us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:21.922325 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:21.922443 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:21.922800 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 11:07:21.922877 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:07:21.933561 16799 raft_consensus.cc:1261] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:21.933717 16661 raft_consensus.cc:1261] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:21.934906 17022 consensus_queue.cc:1035] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:21.935331 17018 consensus_queue.cc:1035] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:21.952276 17025 mvcc.cc:205] Tried to move back new op lower bound from 7333209874146238464 to 7333209874019106816. Current Snapshot: MvccSnapshot[applied={T|T < 7333209874146238464 or (T in {7333209874146238464})}]
I20260925 11:07:21.964800 16502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:21.965189 16502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:21.969187 16502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Sending DeleteTablet for 3 replicas of tablet f6a9b0461eb6458ab0d880042a91bd9f
I20260925 11:07:21.970597 16918 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:21 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50710
I20260925 11:07:21.970625 16779 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:21 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260925 11:07:21.970759 16639 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a9b0461eb6458ab0d880042a91bd9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:21 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:39758
I20260925 11:07:21.974593 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:21.976890 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:21.982653 17037 tablet_replica.cc:330] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: stopping tablet replica
I20260925 11:07:21.983183 17037 raft_consensus.cc:2229] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:21.983798 17037 raft_consensus.cc:2258] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:21.986047 17037 ts_tablet_manager.cc:1905] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:21.994292 17037 ts_tablet_manager.cc:1918] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:21.994570 17037 log.cc:1196] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/f6a9b0461eb6458ab0d880042a91bd9f
I20260925 11:07:21.995242 17037 ts_tablet_manager.cc:1939] T f6a9b0461eb6458ab0d880042a91bd9f P d7d390a712504015a95551210af4874f: Deleting consensus metadata
I20260925 11:07:21.995232 16917 tablet_service.cc:1467] Processing CreateTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 (DEFAULT_TABLE table=test-table [id=6a83c199631a4632ba20f261c96eae74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:21.995697 16639 tablet_service.cc:1467] Processing CreateTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 (DEFAULT_TABLE table=test-table [id=6a83c199631a4632ba20f261c96eae74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:21.995952 16917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f283f7dd113402daf85d7d04a7b0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:21.996163 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f283f7dd113402daf85d7d04a7b0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:21.997601 16488 catalog_manager.cc:4841] TS d7d390a712504015a95551210af4874f (127.8.181.1:40487): tablet f6a9b0461eb6458ab0d880042a91bd9f (table test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]) successfully deleted
I20260925 11:07:21.999405 16778 tablet_service.cc:1467] Processing CreateTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 (DEFAULT_TABLE table=test-table [id=6a83c199631a4632ba20f261c96eae74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.000039 16778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f283f7dd113402daf85d7d04a7b0266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.004536 17036 tablet_replica.cc:330] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: stopping tablet replica
I20260925 11:07:22.005100 17036 raft_consensus.cc:2229] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:22.005715 17013 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Bootstrap starting.
I20260925 11:07:22.005862 17036 raft_consensus.cc:2258] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.007146 17035 tablet_replica.cc:330] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: stopping tablet replica
I20260925 11:07:22.007598 17013 tablet_bootstrap.cc:654] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.007627 17035 raft_consensus.cc:2229] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:22.008162 17035 raft_consensus.cc:2258] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.010216 17036 ts_tablet_manager.cc:1905] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.012548 17015 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Bootstrap starting.
I20260925 11:07:22.014497 17015 tablet_bootstrap.cc:654] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.018687 17036 ts_tablet_manager.cc:1918] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:22.018908 17036 log.cc:1196] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/f6a9b0461eb6458ab0d880042a91bd9f
I20260925 11:07:22.019518 17036 ts_tablet_manager.cc:1939] T f6a9b0461eb6458ab0d880042a91bd9f P f2988a0a465645819fddcf22cfc095ad: Deleting consensus metadata
I20260925 11:07:22.019944 17012 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Bootstrap starting.
I20260925 11:07:22.023419 17012 tablet_bootstrap.cc:654] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.023530 17035 ts_tablet_manager.cc:1905] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.030949 17015 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: No bootstrap required, opened a new log
I20260925 11:07:22.031258 17015 ts_tablet_manager.cc:1397] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20260925 11:07:22.032320 17015 raft_consensus.cc:348] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.032593 17015 raft_consensus.cc:374] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.032702 17015 raft_consensus.cc:729] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5bd4c8b0d469b93805e9b8bfa7735, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.033051 17015 consensus_queue.cc:260] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.033310 17013 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: No bootstrap required, opened a new log
I20260925 11:07:22.033543 17013 ts_tablet_manager.cc:1397] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.007s
I20260925 11:07:22.033740 17035 ts_tablet_manager.cc:1918] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:22.033921 17035 log.cc:1196] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/f6a9b0461eb6458ab0d880042a91bd9f
I20260925 11:07:22.034590 17035 ts_tablet_manager.cc:1939] T f6a9b0461eb6458ab0d880042a91bd9f P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting consensus metadata
I20260925 11:07:22.034761 17013 raft_consensus.cc:348] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.035156 17013 raft_consensus.cc:374] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.035750 16486 catalog_manager.cc:4841] TS f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663): tablet f6a9b0461eb6458ab0d880042a91bd9f (table test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]) successfully deleted
I20260925 11:07:22.035302 17013 raft_consensus.cc:729] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d390a712504015a95551210af4874f, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.036499 16488 catalog_manager.cc:4841] TS bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847): tablet f6a9b0461eb6458ab0d880042a91bd9f (table test-table [id=0c2ce46664cb4fbb814ced4fcf802b63]) successfully deleted
I20260925 11:07:22.036692 17015 ts_tablet_manager.cc:1428] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 11:07:22.036931 17013 consensus_queue.cc:260] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.040550 17013 ts_tablet_manager.cc:1428] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260925 11:07:22.043269 17012 tablet_bootstrap.cc:492] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: No bootstrap required, opened a new log
I20260925 11:07:22.043486 17012 ts_tablet_manager.cc:1397] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.001s
I20260925 11:07:22.044335 17012 raft_consensus.cc:348] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.044618 17012 raft_consensus.cc:374] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.044728 17012 raft_consensus.cc:729] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.045047 17012 consensus_queue.cc:260] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.045746 17012 ts_tablet_manager.cc:1428] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20260925 11:07:22.059367 16710 tablet.cc:2369] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:22.062425 17030 raft_consensus.cc:482] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:22.062820 17030 raft_consensus.cc:504] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.063700 17030 leader_election.cc:288] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d390a712504015a95551210af4874f (127.8.181.1:40487), bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847)
I20260925 11:07:22.064314 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f283f7dd113402daf85d7d04a7b0266" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f" is_pre_election: true
I20260925 11:07:22.064441 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f283f7dd113402daf85d7d04a7b0266" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" is_pre_election: true
I20260925 11:07:22.064580 16661 raft_consensus.cc:2454] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:22.064731 16799 raft_consensus.cc:2454] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:22.065106 16871 leader_election.cc:302] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d390a712504015a95551210af4874f, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.065485 17030 raft_consensus.cc:2781] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:22.065619 17030 raft_consensus.cc:482] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:22.065707 17030 raft_consensus.cc:3037] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.069211 17030 raft_consensus.cc:504] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.070013 17030 leader_election.cc:288] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Requested vote from peers d7d390a712504015a95551210af4874f (127.8.181.1:40487), bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847)
I20260925 11:07:22.070792 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f283f7dd113402daf85d7d04a7b0266" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f"
I20260925 11:07:22.071035 16661 raft_consensus.cc:3037] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.070968 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f283f7dd113402daf85d7d04a7b0266" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
I20260925 11:07:22.071240 16799 raft_consensus.cc:3037] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.073295 16799 raft_consensus.cc:2454] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:22.073954 16871 leader_election.cc:302] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb5bd4c8b0d469b93805e9b8bfa7735, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.074280 16661 raft_consensus.cc:2454] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:22.074329 17030 raft_consensus.cc:2781] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:22.074955 17030 raft_consensus.cc:686] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Becoming Leader. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Running, Role: LEADER
I20260925 11:07:22.075286 17030 consensus_queue.cc:237] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.078555 16502 catalog_manager.cc:5495] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f2988a0a465645819fddcf22cfc095ad (127.8.181.3). New cstate: current_term: 1 leader_uuid: "f2988a0a465645819fddcf22cfc095ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } health_report { overall_health: HEALTHY } } }
Sep 25 11:07:22 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334439, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.62@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:22.142815 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.125845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.8.181.62:46409)
0925 11:07:22.126102 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.126106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.126816 (+ 710us) server_negotiation.cc:407] Connection header received
0925 11:07:22.127042 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.127049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.127111 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.127227 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:22.128857 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.129791 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.131481 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.131914 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.132065 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.136037 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:22.136042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:22.136045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:22.136075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:22.139255 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:22.140482 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:22.140489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:22.140494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:22.140679 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:22.141078 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:22.141083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:22.141086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:22.141434 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:22.141597 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.142096 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.142414 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":111}
I20260925 11:07:22.142838 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.125796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:46409 (local address 127.0.0.1:58348)
0925 11:07:22.126692 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.126714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.126906 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.127257 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.127261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.128015 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:22.128306 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.128316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.129933 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.129941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.131301 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.131317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.132106 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.132112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.132307 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.133073 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:22.133096 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:22.135814 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:22.139434 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:22.139453 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:22.139462 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:22.140296 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:22.140805 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:22.140810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:22.140814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:22.140959 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:22.141573 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:22.141582 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:22.141766 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.141932 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.142276 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":555,"threads_started":1}
I20260925 11:07:22.149889 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:58348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 11:07:22.151392 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:22.160063 16779 tablet_service.cc:1467] Processing CreateTablet for tablet e616a97cba5349e6a351a9998b4eb8e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c259dffc0b942f98ff2dd60464c0261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:07:22.160051 16639 tablet_service.cc:1467] Processing CreateTablet for tablet e616a97cba5349e6a351a9998b4eb8e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c259dffc0b942f98ff2dd60464c0261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:07:22.160698 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616a97cba5349e6a351a9998b4eb8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.160712 16779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616a97cba5349e6a351a9998b4eb8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.160624 16917 tablet_service.cc:1467] Processing CreateTablet for tablet e616a97cba5349e6a351a9998b4eb8e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c259dffc0b942f98ff2dd60464c0261]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 11:07:22.161309 16917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e616a97cba5349e6a351a9998b4eb8e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.168253 17015 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: Bootstrap starting.
I20260925 11:07:22.169200 17013 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: Bootstrap starting.
I20260925 11:07:22.170096 17015 tablet_bootstrap.cc:654] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.170646 17013 tablet_bootstrap.cc:654] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.171603 17012 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: Bootstrap starting.
I20260925 11:07:22.173492 17012 tablet_bootstrap.cc:654] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.174453 17015 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: No bootstrap required, opened a new log
I20260925 11:07:22.174774 17015 ts_tablet_manager.cc:1397] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260925 11:07:22.176124 17015 raft_consensus.cc:348] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.178119 17015 raft_consensus.cc:374] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.178277 17015 raft_consensus.cc:729] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5bd4c8b0d469b93805e9b8bfa7735, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.178754 17015 consensus_queue.cc:260] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.179569 17015 ts_tablet_manager.cc:1428] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 11:07:22.179414 17020 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.179826 17020 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.181841 17012 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: No bootstrap required, opened a new log
I20260925 11:07:22.182051 17012 ts_tablet_manager.cc:1397] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 11:07:22.182910 17013 tablet_bootstrap.cc:492] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: No bootstrap required, opened a new log
I20260925 11:07:22.183109 17013 ts_tablet_manager.cc:1397] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260925 11:07:22.182998 17012 raft_consensus.cc:348] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.183267 17012 raft_consensus.cc:374] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.183384 17012 raft_consensus.cc:729] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.183630 17012 consensus_queue.cc:260] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.183897 17013 raft_consensus.cc:348] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.184136 17013 raft_consensus.cc:374] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.184207 17013 raft_consensus.cc:729] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d390a712504015a95551210af4874f, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.184273 17012 ts_tablet_manager.cc:1428] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 11:07:22.184165 17030 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.184485 17030 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.184451 17013 consensus_queue.cc:260] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.185506 17019 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.185858 17019 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.186430 17013 ts_tablet_manager.cc:1428] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 11:07:22.258060 17019 raft_consensus.cc:482] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:22.258338 17019 raft_consensus.cc:504] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.260663 17019 leader_election.cc:288] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663)
I20260925 11:07:22.268947 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.261442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:50779 (local address 127.8.181.3:34663)
0925 11:07:22.261664 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.261670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.262072 (+ 402us) server_negotiation.cc:407] Connection header received
0925 11:07:22.262421 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.262430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.262537 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.262630 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:22.263384 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.264256 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.266983 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.267874 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.267959 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.268042 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.268492 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.268629 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:07:22.269053 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.260867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:34663 (local address 127.8.181.1:50779)
0925 11:07:22.261936 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.261959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.262220 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.262671 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.262675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.262707 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:22.263164 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.263177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.264414 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.264417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.266764 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.266782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.268198 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.268203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.268319 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.268342 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.268461 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.268617 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":33,"thread_start_us":749,"threads_started":1}
I20260925 11:07:22.269796 17008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.261212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:47781 (local address 127.8.181.2:46847)
0925 11:07:22.261417 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.261424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.262497 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 11:07:22.262667 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.262672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.262752 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.262852 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:22.263586 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.264712 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.267075 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.268369 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.268505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.268619 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.269342 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.269491 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:07:22.270017 16938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616a97cba5349e6a351a9998b4eb8e9" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2988a0a465645819fddcf22cfc095ad" is_pre_election: true
I20260925 11:07:22.270458 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.260944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46847 (local address 127.8.181.1:47781)
0925 11:07:22.262340 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.262364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.262523 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.262889 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.262893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.262913 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:22.263379 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.263390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.264878 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.264883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.266907 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.266922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.268788 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.268795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.268976 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.269017 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.269874 (+ 857us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.270037 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":27,"thread_start_us":1090,"threads_started":1}
I20260925 11:07:22.270625 16938 raft_consensus.cc:2454] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 0.
I20260925 11:07:22.271266 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616a97cba5349e6a351a9998b4eb8e9" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" is_pre_election: true
I20260925 11:07:22.271371 16589 leader_election.cc:302] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d390a712504015a95551210af4874f, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.271559 16799 raft_consensus.cc:2454] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 0.
I20260925 11:07:22.271832 17019 raft_consensus.cc:2781] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:22.271975 17019 raft_consensus.cc:482] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:22.272081 17019 raft_consensus.cc:3037] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.275681 17019 raft_consensus.cc:504] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.276451 17019 leader_election.cc:288] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 election: Requested vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663)
I20260925 11:07:22.277032 16799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616a97cba5349e6a351a9998b4eb8e9" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
I20260925 11:07:22.277192 16938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e616a97cba5349e6a351a9998b4eb8e9" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2988a0a465645819fddcf22cfc095ad"
I20260925 11:07:22.277293 16799 raft_consensus.cc:3037] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.277437 16938 raft_consensus.cc:3037] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.279440 16799 raft_consensus.cc:2454] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 1.
I20260925 11:07:22.279968 16591 leader_election.cc:302] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb5bd4c8b0d469b93805e9b8bfa7735, d7d390a712504015a95551210af4874f; no voters:
I20260925 11:07:22.281217 16938 raft_consensus.cc:2454] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 1.
I20260925 11:07:22.281773 17019 raft_consensus.cc:2781] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:22.282428 17019 raft_consensus.cc:686] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [term 1 LEADER]: Becoming Leader. State: Replica: d7d390a712504015a95551210af4874f, State: Running, Role: LEADER
I20260925 11:07:22.282996 17019 consensus_queue.cc:237] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.285696 17053 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: TxnStatusTablet state changed. Reason: New leader d7d390a712504015a95551210af4874f. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.286170 17053 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:07:22.286921 17055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 11:07:22.287156 17055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 11:07:22.287734 16502 catalog_manager.cc:5495] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d390a712504015a95551210af4874f (127.8.181.1). New cstate: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } health_report { overall_health: UNKNOWN } } }
I20260925 11:07:22.318150 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.309518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.8.181.1:40487)
0925 11:07:22.309815 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.309822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.309942 (+ 120us) server_negotiation.cc:407] Connection header received
0925 11:07:22.310293 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.310299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.310437 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.310753 (+ 316us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:22.311363 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.312723 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.314959 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.315625 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.315774 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.316497 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.317332 (+ 835us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.317499 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.317685 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.317782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 11:07:22.318197 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.309389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40487 (local address 127.0.0.1:34428)
0925 11:07:22.309814 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.309837 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.310029 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.310958 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.310964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.310978 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:07:22.311182 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.311194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.312899 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.312906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.314730 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.314750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.315824 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.315832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.316126 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.316222 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.317496 (+ 1274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.317530 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.317645 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.317857 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260925 11:07:22.323371 16799 raft_consensus.cc:1261] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Refusing update from remote peer d7d390a712504015a95551210af4874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.323721 16938 raft_consensus.cc:1261] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Refusing update from remote peer d7d390a712504015a95551210af4874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.324311 17053 consensus_queue.cc:1035] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.324699 17019 consensus_queue.cc:1035] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.332525 17030 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: TxnStatusTablet state changed. Reason: New leader d7d390a712504015a95551210af4874f. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.332964 17030 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.335873 17053 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.336162 17053 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:07:22.336364 17027 mvcc.cc:205] Tried to move back new op lower bound from 7333209875747893248 to 7333209875595001856. Current Snapshot: MvccSnapshot[applied={T|T < 7333209875747893248 or (T in {7333209875747893248})}]
I20260925 11:07:22.339938 17030 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.340312 17030 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P f2988a0a465645819fddcf22cfc095ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.344167 17020 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: TxnStatusTablet state changed. Reason: New leader d7d390a712504015a95551210af4874f. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.344501 17020 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.346244 17019 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.346633 17019 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P d7d390a712504015a95551210af4874f: This TxnStatusTablet replica's current role is: LEADER
I20260925 11:07:22.355443 17020 tablet_replica.cc:439] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } } }
I20260925 11:07:22.355758 17020 tablet_replica.cc:442] T e616a97cba5349e6a351a9998b4eb8e9 P bbb5bd4c8b0d469b93805e9b8bfa7735: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 11:07:22.381435 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.374419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57465 (local address 127.8.181.1:40487)
0925 11:07:22.374662 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.374670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.375459 (+ 789us) server_negotiation.cc:407] Connection header received
0925 11:07:22.375706 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.375715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.375803 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.375912 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:22.376513 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.377311 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.379289 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.379884 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.380039 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.380535 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.380691 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.380796 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.380961 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.381048 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 11:07:22.381497 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.374427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40487 (local address 127.8.181.3:57465)
0925 11:07:22.375331 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.375347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.375550 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.375926 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.375931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.375946 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:07:22.376330 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.376339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.377460 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.377467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.379098 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.379113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.380087 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.380095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.380274 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.380339 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.380814 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.380829 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.380934 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.381092 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":499,"threads_started":1}
I20260925 11:07:22.394906 16799 raft_consensus.cc:1261] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.395011 16661 raft_consensus.cc:1261] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.395572 17022 consensus_queue.cc:1035] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.396135 17022 consensus_queue.cc:1035] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.467928 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.453328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:43475 (local address 127.8.181.3:34663)
0925 11:07:22.453529 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.453540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.460215 (+ 6675us) server_negotiation.cc:407] Connection header received
0925 11:07:22.460442 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.460448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.460544 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.460675 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:22.461435 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.462675 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.465300 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.466060 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.466180 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.466975 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.467144 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.467222 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.467467 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.467562 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:07:22.468801 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.452949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:34663 (local address 127.8.181.1:43475)
0925 11:07:22.460055 (+ 7106us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:22.460079 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:22.460254 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:22.460708 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:22.460713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:22.460726 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:07:22.461209 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.461222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.462874 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.462880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.465084 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:22.465105 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.466405 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.466412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:22.466637 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.466733 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.467310 (+ 577us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.467319 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 11:07:22.468175 (+ 856us) client_negotiation.cc:240] Negotiation successful
0925 11:07:22.468354 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6854,"thread_start_us":6634,"threads_started":1}
I20260925 11:07:22.521381 16502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:22.521682 16502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:22.526420 16502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Sending DeleteTablet for 3 replicas of tablet 4f283f7dd113402daf85d7d04a7b0266
I20260925 11:07:22.528327 16779 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260925 11:07:22.528936 17035 tablet_replica.cc:330] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: stopping tablet replica
I20260925 11:07:22.529206 17035 raft_consensus.cc:2229] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:22.529456 17035 raft_consensus.cc:2258] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.530632 16917 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50710
I20260925 11:07:22.531327 17036 tablet_replica.cc:330] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: stopping tablet replica
I20260925 11:07:22.531754 17036 raft_consensus.cc:2229] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:22.532390 17036 raft_consensus.cc:2258] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.533845 17036 ts_tablet_manager.cc:1905] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.533944 17035 ts_tablet_manager.cc:1905] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.534726 16639 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f283f7dd113402daf85d7d04a7b0266 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:39758
I20260925 11:07:22.543565 17036 ts_tablet_manager.cc:1918] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:22.543964 17036 log.cc:1196] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/4f283f7dd113402daf85d7d04a7b0266
I20260925 11:07:22.544493 17036 ts_tablet_manager.cc:1939] T 4f283f7dd113402daf85d7d04a7b0266 P f2988a0a465645819fddcf22cfc095ad: Deleting consensus metadata
I20260925 11:07:22.545820 16486 catalog_manager.cc:4841] TS f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663): tablet 4f283f7dd113402daf85d7d04a7b0266 (table test-table [id=6a83c199631a4632ba20f261c96eae74]) successfully deleted
I20260925 11:07:22.546720 17069 tablet_replica.cc:330] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: stopping tablet replica
I20260925 11:07:22.547037 17069 raft_consensus.cc:2229] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:22.547489 17069 raft_consensus.cc:2258] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.549763 17035 ts_tablet_manager.cc:1918] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:22.549937 17035 log.cc:1196] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/4f283f7dd113402daf85d7d04a7b0266
I20260925 11:07:22.550405 17035 ts_tablet_manager.cc:1939] T 4f283f7dd113402daf85d7d04a7b0266 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting consensus metadata
I20260925 11:07:22.551805 16488 catalog_manager.cc:4841] TS bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847): tablet 4f283f7dd113402daf85d7d04a7b0266 (table test-table [id=6a83c199631a4632ba20f261c96eae74]) successfully deleted
I20260925 11:07:22.552855 17069 ts_tablet_manager.cc:1905] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.569389 17069 ts_tablet_manager.cc:1918] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:22.569581 17069 log.cc:1196] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/4f283f7dd113402daf85d7d04a7b0266
I20260925 11:07:22.570173 17069 ts_tablet_manager.cc:1939] T 4f283f7dd113402daf85d7d04a7b0266 P d7d390a712504015a95551210af4874f: Deleting consensus metadata
I20260925 11:07:22.571580 16488 catalog_manager.cc:4841] TS d7d390a712504015a95551210af4874f (127.8.181.1:40487): tablet 4f283f7dd113402daf85d7d04a7b0266 (table test-table [id=6a83c199631a4632ba20f261c96eae74]) successfully deleted
Sep 25 11:07:22 dist-test-slave-6042 krb5kdc[16457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 11:07:22 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:22 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:22.627701 16737 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:22.609224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.8.181.62:46409)
0925 11:07:22.609462 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.609471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.610000 (+ 529us) server_negotiation.cc:407] Connection header received
0925 11:07:22.610203 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.610209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.610312 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.610429 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:22.611983 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.613177 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.614996 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.615685 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.615840 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.627414 (+ 11574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36962: BlockingRecv error: recv got EOF from 127.0.0.1:36962 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 11:07:22 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM
I20260925 11:07:22.647612 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.630891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.8.181.62:46409)
0925 11:07:22.631070 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.631076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.631372 (+ 296us) server_negotiation.cc:407] Connection header received
0925 11:07:22.631518 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.631525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.631607 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.631733 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:22.633002 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.634164 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.635650 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.636358 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.636500 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.640675 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:22.640683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:22.640699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:22.640753 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:22.644384 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:22.645194 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:22.645199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:22.645203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:22.645378 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:22.645879 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:22.645884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:22.645888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:22.646268 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:22.646544 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.646978 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.647240 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:07:22.652173 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:22.653061 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:22.660892 16779 tablet_service.cc:1467] Processing CreateTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 (DEFAULT_TABLE table=test-table [id=8b775da314764c4fb0ffee64cef03d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.660997 16917 tablet_service.cc:1467] Processing CreateTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 (DEFAULT_TABLE table=test-table [id=8b775da314764c4fb0ffee64cef03d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.661386 16779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afa12c3fdd40ae804d87c50bd85f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.661414 16917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afa12c3fdd40ae804d87c50bd85f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.663282 16639 tablet_service.cc:1467] Processing CreateTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 (DEFAULT_TABLE table=test-table [id=8b775da314764c4fb0ffee64cef03d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.663710 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afa12c3fdd40ae804d87c50bd85f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.668062 17015 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Bootstrap starting.
I20260925 11:07:22.668277 17012 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Bootstrap starting.
I20260925 11:07:22.669763 17012 tablet_bootstrap.cc:654] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.669889 17015 tablet_bootstrap.cc:654] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.677881 17013 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Bootstrap starting.
I20260925 11:07:22.679909 17013 tablet_bootstrap.cc:654] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.682116 17015 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: No bootstrap required, opened a new log
I20260925 11:07:22.682360 17015 ts_tablet_manager.cc:1397] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260925 11:07:22.682384 17012 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: No bootstrap required, opened a new log
I20260925 11:07:22.682667 17012 ts_tablet_manager.cc:1397] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260925 11:07:22.683269 17015 raft_consensus.cc:348] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.683552 17015 raft_consensus.cc:374] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.683662 17015 raft_consensus.cc:729] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5bd4c8b0d469b93805e9b8bfa7735, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.683513 17012 raft_consensus.cc:348] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.683991 17012 raft_consensus.cc:374] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.683933 17015 consensus_queue.cc:260] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.684207 17012 raft_consensus.cc:729] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.684469 17012 consensus_queue.cc:260] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.685156 17012 ts_tablet_manager.cc:1428] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 11:07:22.686560 17015 ts_tablet_manager.cc:1428] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 11:07:22.690097 17013 tablet_bootstrap.cc:492] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: No bootstrap required, opened a new log
I20260925 11:07:22.690294 17013 ts_tablet_manager.cc:1397] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20260925 11:07:22.691072 17013 raft_consensus.cc:348] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.691358 17013 raft_consensus.cc:374] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.691473 17013 raft_consensus.cc:729] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d390a712504015a95551210af4874f, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.691778 17013 consensus_queue.cc:260] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.692497 17013 ts_tablet_manager.cc:1428] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 11:07:22.746120 17030 raft_consensus.cc:482] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:22.746400 17030 raft_consensus.cc:504] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.747061 17030 leader_election.cc:288] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d390a712504015a95551210af4874f (127.8.181.1:40487), bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847)
I20260925 11:07:22.747766 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afa12c3fdd40ae804d87c50bd85f40" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" is_pre_election: true
I20260925 11:07:22.747786 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afa12c3fdd40ae804d87c50bd85f40" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f" is_pre_election: true
I20260925 11:07:22.748034 16798 raft_consensus.cc:2454] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:22.748055 16661 raft_consensus.cc:2454] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 0.
I20260925 11:07:22.748466 16871 leader_election.cc:302] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb5bd4c8b0d469b93805e9b8bfa7735, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.748818 17030 raft_consensus.cc:2781] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:22.748930 17030 raft_consensus.cc:482] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:22.749037 17030 raft_consensus.cc:3037] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.751251 17030 raft_consensus.cc:504] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.751839 17030 leader_election.cc:288] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Requested vote from peers d7d390a712504015a95551210af4874f (127.8.181.1:40487), bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847)
I20260925 11:07:22.752488 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afa12c3fdd40ae804d87c50bd85f40" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
I20260925 11:07:22.752534 16661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afa12c3fdd40ae804d87c50bd85f40" candidate_uuid: "f2988a0a465645819fddcf22cfc095ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d390a712504015a95551210af4874f"
I20260925 11:07:22.752723 16798 raft_consensus.cc:3037] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.752774 16661 raft_consensus.cc:3037] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.754920 16798 raft_consensus.cc:2454] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:22.755422 16871 leader_election.cc:302] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbb5bd4c8b0d469b93805e9b8bfa7735, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.755698 17030 raft_consensus.cc:2781] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:22.755882 17030 raft_consensus.cc:686] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Becoming Leader. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Running, Role: LEADER
I20260925 11:07:22.756155 16661 raft_consensus.cc:2454] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2988a0a465645819fddcf22cfc095ad in term 1.
I20260925 11:07:22.756178 17030 consensus_queue.cc:237] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.759253 16502 catalog_manager.cc:5495] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f2988a0a465645819fddcf22cfc095ad (127.8.181.3). New cstate: current_term: 1 leader_uuid: "f2988a0a465645819fddcf22cfc095ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } health_report { overall_health: HEALTHY } } }
I20260925 11:07:22.793097 17009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:22.787083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.8.181.3:34663)
0925 11:07:22.787283 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:22.787289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:22.787312 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:07:22.787475 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:22.787481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:22.787572 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:22.787704 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:22.788210 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.789344 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.790842 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:22.791534 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:22.791675 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:22.792111 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:22.792285 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:22.792382 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:22.792670 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 11:07:22.792759 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:07:22.796383 16661 raft_consensus.cc:1261] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.796442 16798 raft_consensus.cc:1261] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Refusing update from remote peer f2988a0a465645819fddcf22cfc095ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:22.797044 17022 consensus_queue.cc:1035] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.797400 17030 consensus_queue.cc:1035] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:22.808475 17026 mvcc.cc:205] Tried to move back new op lower bound from 7333209877687828480 to 7333209877532209152. Current Snapshot: MvccSnapshot[applied={T|T < 7333209877687828480}]
I20260925 11:07:22.812476 16502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:22.812758 16502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:22.816133 16502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Sending DeleteTablet for 3 replicas of tablet c4afa12c3fdd40ae804d87c50bd85f40
I20260925 11:07:22.817293 16779 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260925 11:07:22.817293 16639 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:39758
I20260925 11:07:22.817652 17035 tablet_replica.cc:330] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: stopping tablet replica
I20260925 11:07:22.817832 17035 raft_consensus.cc:2229] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:22.818063 17035 raft_consensus.cc:2258] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.818328 17069 tablet_replica.cc:330] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: stopping tablet replica
I20260925 11:07:22.818557 17069 raft_consensus.cc:2229] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:22.818778 17069 raft_consensus.cc:2258] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.819988 17035 ts_tablet_manager.cc:1905] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.820104 16917 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afa12c3fdd40ae804d87c50bd85f40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:22 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50710
I20260925 11:07:22.820430 17036 tablet_replica.cc:330] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: stopping tablet replica
I20260925 11:07:22.820613 17036 raft_consensus.cc:2229] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:22.820952 17036 raft_consensus.cc:2258] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:22.821928 17036 ts_tablet_manager.cc:1905] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.827149 17035 ts_tablet_manager.cc:1918] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:22.827164 17036 ts_tablet_manager.cc:1918] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:22.827332 17036 log.cc:1196] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c4afa12c3fdd40ae804d87c50bd85f40
I20260925 11:07:22.827361 17035 log.cc:1196] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c4afa12c3fdd40ae804d87c50bd85f40
I20260925 11:07:22.827720 17036 ts_tablet_manager.cc:1939] T c4afa12c3fdd40ae804d87c50bd85f40 P f2988a0a465645819fddcf22cfc095ad: Deleting consensus metadata
I20260925 11:07:22.827803 17035 ts_tablet_manager.cc:1939] T c4afa12c3fdd40ae804d87c50bd85f40 P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting consensus metadata
I20260925 11:07:22.820855 16502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:22.828590 16502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:22.828944 16488 catalog_manager.cc:4841] TS bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847): tablet c4afa12c3fdd40ae804d87c50bd85f40 (table test-table [id=8b775da314764c4fb0ffee64cef03d86]) successfully deleted
I20260925 11:07:22.829995 17069 ts_tablet_manager.cc:1905] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:22.831598 16486 catalog_manager.cc:4841] TS f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663): tablet c4afa12c3fdd40ae804d87c50bd85f40 (table test-table [id=8b775da314764c4fb0ffee64cef03d86]) successfully deleted
I20260925 11:07:22.836270 17069 ts_tablet_manager.cc:1918] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:22.836448 17069 log.cc:1196] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c4afa12c3fdd40ae804d87c50bd85f40
I20260925 11:07:22.837033 17069 ts_tablet_manager.cc:1939] T c4afa12c3fdd40ae804d87c50bd85f40 P d7d390a712504015a95551210af4874f: Deleting consensus metadata
I20260925 11:07:22.838184 16488 catalog_manager.cc:4841] TS d7d390a712504015a95551210af4874f (127.8.181.1:40487): tablet c4afa12c3fdd40ae804d87c50bd85f40 (table test-table [id=8b775da314764c4fb0ffee64cef03d86]) successfully deleted
I20260925 11:07:22.840616 16917 tablet_service.cc:1467] Processing CreateTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (DEFAULT_TABLE table=test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.840628 16639 tablet_service.cc:1467] Processing CreateTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (DEFAULT_TABLE table=test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.841091 16917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.841141 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.846314 17012 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Bootstrap starting.
I20260925 11:07:22.848016 17013 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Bootstrap starting.
I20260925 11:07:22.848111 17012 tablet_bootstrap.cc:654] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.849493 16779 tablet_service.cc:1467] Processing CreateTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (DEFAULT_TABLE table=test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:22.849804 17013 tablet_bootstrap.cc:654] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.849994 16779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:22.852298 17013 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: No bootstrap required, opened a new log
I20260925 11:07:22.852487 17013 ts_tablet_manager.cc:1397] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 11:07:22.853284 17013 raft_consensus.cc:348] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.853576 17013 raft_consensus.cc:374] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.853654 17013 raft_consensus.cc:729] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d390a712504015a95551210af4874f, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.853937 17013 consensus_queue.cc:260] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.855749 17013 ts_tablet_manager.cc:1428] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 11:07:22.857149 17015 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Bootstrap starting.
I20260925 11:07:22.858897 17015 tablet_bootstrap.cc:654] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:22.861550 17015 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: No bootstrap required, opened a new log
I20260925 11:07:22.861745 17015 ts_tablet_manager.cc:1397] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 11:07:22.862401 17015 raft_consensus.cc:348] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.862731 17015 raft_consensus.cc:374] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.862838 17015 raft_consensus.cc:729] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5bd4c8b0d469b93805e9b8bfa7735, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.863085 17015 consensus_queue.cc:260] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.863243 17012 tablet_bootstrap.cc:492] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: No bootstrap required, opened a new log
I20260925 11:07:22.863444 17012 ts_tablet_manager.cc:1397] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.006s
I20260925 11:07:22.863634 17015 ts_tablet_manager.cc:1428] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 11:07:22.864305 17012 raft_consensus.cc:348] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.864593 17012 raft_consensus.cc:374] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:22.864697 17012 raft_consensus.cc:729] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2988a0a465645819fddcf22cfc095ad, State: Initialized, Role: FOLLOWER
I20260925 11:07:22.864939 17012 consensus_queue.cc:260] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.865459 17012 ts_tablet_manager.cc:1428] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 11:07:22.971796 17053 raft_consensus.cc:482] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:22.972086 17053 raft_consensus.cc:504] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.972743 17053 leader_election.cc:288] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663)
I20260925 11:07:22.973452 16937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724c8d41fb6c46d5b3c2a88ae2d39b0a" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2988a0a465645819fddcf22cfc095ad" is_pre_election: true
I20260925 11:07:22.973480 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724c8d41fb6c46d5b3c2a88ae2d39b0a" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" is_pre_election: true
I20260925 11:07:22.973735 16937 raft_consensus.cc:2454] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 0.
I20260925 11:07:22.973790 16798 raft_consensus.cc:2454] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 0.
I20260925 11:07:22.974274 16589 leader_election.cc:302] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d390a712504015a95551210af4874f, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.974642 17053 raft_consensus.cc:2781] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:22.974751 17053 raft_consensus.cc:482] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:22.974838 17053 raft_consensus.cc:3037] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.976908 17053 raft_consensus.cc:504] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.977576 17053 leader_election.cc:288] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 election: Requested vote from peers bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847), f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663)
I20260925 11:07:22.978302 16798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724c8d41fb6c46d5b3c2a88ae2d39b0a" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735"
I20260925 11:07:22.978330 16937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "724c8d41fb6c46d5b3c2a88ae2d39b0a" candidate_uuid: "d7d390a712504015a95551210af4874f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2988a0a465645819fddcf22cfc095ad"
I20260925 11:07:22.978595 16937 raft_consensus.cc:3037] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.978595 16798 raft_consensus.cc:3037] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:22.981637 16798 raft_consensus.cc:2454] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 1.
I20260925 11:07:22.981637 16937 raft_consensus.cc:2454] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d390a712504015a95551210af4874f in term 1.
I20260925 11:07:22.982151 16589 leader_election.cc:302] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7d390a712504015a95551210af4874f, f2988a0a465645819fddcf22cfc095ad; no voters:
I20260925 11:07:22.982720 17053 raft_consensus.cc:2781] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:22.983024 17053 raft_consensus.cc:686] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 1 LEADER]: Becoming Leader. State: Replica: d7d390a712504015a95551210af4874f, State: Running, Role: LEADER
I20260925 11:07:22.983341 17053 consensus_queue.cc:237] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } }
I20260925 11:07:22.986021 16499 catalog_manager.cc:5495] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d390a712504015a95551210af4874f (127.8.181.1). New cstate: current_term: 1 leader_uuid: "d7d390a712504015a95551210af4874f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d390a712504015a95551210af4874f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 } health_report { overall_health: UNKNOWN } } }
I20260925 11:07:23.030692 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.025201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.8.181.1:40487)
0925 11:07:23.025813 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:23.025819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:23.025842 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:07:23.025896 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:23.025900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:23.025959 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:23.026087 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:23.026562 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.027315 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.028720 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.029172 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.029314 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.029825 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:23.029954 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:23.030039 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:23.030175 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 11:07:23.030254 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":260,"threads_started":1}
I20260925 11:07:23.053694 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.046202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:54325 (local address 127.8.181.1:40487)
0925 11:07:23.046387 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:23.046395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:23.047296 (+ 901us) server_negotiation.cc:407] Connection header received
0925 11:07:23.047475 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:23.047482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:23.047571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:23.047679 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:23.048260 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.049156 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.051096 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.051765 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.051910 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.052633 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:23.052839 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:23.052954 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:23.053159 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 11:07:23.053254 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260925 11:07:23.053790 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.046202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:40487 (local address 127.8.181.1:54325)
0925 11:07:23.047160 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:23.047181 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:23.047342 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:23.047718 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:23.047723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:23.047735 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 11:07:23.048073 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:23.048085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.049332 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.049336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:23.050927 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:23.050940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.052082 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.052092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:23.052307 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.052374 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 11:07:23.052969 (+ 595us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 11:07:23.052986 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 11:07:23.053107 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 11:07:23.053295 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":433,"threads_started":1}
I20260925 11:07:23.062156 16937 raft_consensus.cc:1261] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Refusing update from remote peer d7d390a712504015a95551210af4874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:23.062165 16798 raft_consensus.cc:1261] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Refusing update from remote peer d7d390a712504015a95551210af4874f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:23.062758 17093 consensus_queue.cc:1035] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb5bd4c8b0d469b93805e9b8bfa7735" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:23.063148 17060 consensus_queue.cc:1035] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2988a0a465645819fddcf22cfc095ad" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 34663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:23.133535 16499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:23.133826 16499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:23.137518 16499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d1d368d267b4286a3c10d5d81cc4530: Sending DeleteTablet for 3 replicas of tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a
I20260925 11:07:23.138592 16779 tablet_service.cc:1514] Processing DeleteTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:23 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:51796
I20260925 11:07:23.138746 16917 tablet_service.cc:1514] Processing DeleteTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:23 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50710
I20260925 11:07:23.138943 17035 tablet_replica.cc:330] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: stopping tablet replica
I20260925 11:07:23.139017 17036 tablet_replica.cc:330] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: stopping tablet replica
I20260925 11:07:23.139153 17035 raft_consensus.cc:2229] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:23.139189 17036 raft_consensus.cc:2229] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:23.139367 17035 raft_consensus.cc:2258] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:23.139390 17036 raft_consensus.cc:2258] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:23.140089 16639 tablet_service.cc:1514] Processing DeleteTablet for tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:23 UTC) from {username='oryx', principal='oryx/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:39758
I20260925 11:07:23.140407 17069 tablet_replica.cc:330] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: stopping tablet replica
I20260925 11:07:23.140604 17069 raft_consensus.cc:2229] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:23.140646 17036 ts_tablet_manager.cc:1905] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:23.140727 17035 ts_tablet_manager.cc:1905] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:23.141223 17069 raft_consensus.cc:2258] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:23.144765 17069 ts_tablet_manager.cc:1905] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:23.146903 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16573
I20260925 11:07:23.147675 17036 ts_tablet_manager.cc:1918] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:23.147675 17035 ts_tablet_manager.cc:1918] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:23.147840 17035 log.cc:1196] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/724c8d41fb6c46d5b3c2a88ae2d39b0a
I20260925 11:07:23.147830 17036 log.cc:1196] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/724c8d41fb6c46d5b3c2a88ae2d39b0a
I20260925 11:07:23.148248 17035 ts_tablet_manager.cc:1939] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P bbb5bd4c8b0d469b93805e9b8bfa7735: Deleting consensus metadata
I20260925 11:07:23.148597 17036 ts_tablet_manager.cc:1939] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P f2988a0a465645819fddcf22cfc095ad: Deleting consensus metadata
I20260925 11:07:23.149802 16486 catalog_manager.cc:4841] TS f2988a0a465645819fddcf22cfc095ad (127.8.181.3:34663): tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (table test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]) successfully deleted
I20260925 11:07:23.150593 16488 catalog_manager.cc:4841] TS bbb5bd4c8b0d469b93805e9b8bfa7735 (127.8.181.2:46847): tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (table test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]) successfully deleted
I20260925 11:07:23.154875 17069 ts_tablet_manager.cc:1918] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 11:07:23.155058 17069 log.cc:1196] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/724c8d41fb6c46d5b3c2a88ae2d39b0a
I20260925 11:07:23.155634 17069 ts_tablet_manager.cc:1939] T 724c8d41fb6c46d5b3c2a88ae2d39b0a P d7d390a712504015a95551210af4874f: Deleting consensus metadata
I20260925 11:07:23.156757 16488 catalog_manager.cc:4841] TS d7d390a712504015a95551210af4874f (127.8.181.1:40487): tablet 724c8d41fb6c46d5b3c2a88ae2d39b0a (table test-table [id=ad0cf6ab2e9e4170966a3b4a2bf3ac92]) successfully deleted
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.1@KRBTEST.COM
I20260925 11:07:23.173102 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.148772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.8.181.1:40487)
0925 11:07:23.148994 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:23.149000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:23.155138 (+ 6138us) server_negotiation.cc:407] Connection header received
0925 11:07:23.155447 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:23.155456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:23.155557 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:23.155704 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:23.157644 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.158463 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.160523 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.161036 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.161176 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.166523 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:23.166552 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:23.166569 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:23.166612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:23.169903 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.170870 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.170878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.170880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.171029 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.171458 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.171462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.171465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.171850 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:23.172207 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:23.172491 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 11:07:23.172733 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:07:23.173785 16704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:23.308346 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16573
I20260925 11:07:23.333135 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16713
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.2@KRBTEST.COM
I20260925 11:07:23.349133 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.333966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51700 (local address 127.8.181.2:46847)
0925 11:07:23.334426 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:23.334431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:23.334447 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:07:23.334555 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:23.334558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:23.334618 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:23.334716 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:23.335939 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.336730 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.337893 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.338343 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.338437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.342925 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:23.342940 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:23.342957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:23.342999 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:23.346161 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.347157 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.347166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.347168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.347302 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.347718 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.347722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.347724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.348092 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:23.348372 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:23.348616 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 11:07:23.348870 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
I20260925 11:07:23.349886 16844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:23.451313 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16713
I20260925 11:07:23.469787 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16853
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[16457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.181.3@KRBTEST.COM
I20260925 11:07:23.488044 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:23.470806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.8.181.3:34663)
0925 11:07:23.471317 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:23.471321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:23.471338 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:23.471396 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:23.471399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:23.471465 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:23.471565 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:23.473231 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.474020 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.475368 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:23.475802 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:23.475909 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:23.481330 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:23.481356 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:23.481372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:23.481418 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:23.484978 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.485840 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.485848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.485851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.486008 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:23.486419 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:23.486423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:23.486425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:23.486937 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:23.487189 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:23.487568 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 11:07:23.487759 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":309,"threads_started":1}
I20260925 11:07:23.488821 16983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:23.616852 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16853
I20260925 11:07:23.640646 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 16470
I20260925 11:07:23.641904 16542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:23.767341 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 16470
2026-09-25T11:07:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5770 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[17111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[17111](info): set up 2 sockets
Sep 25 11:07:23 dist-test-slave-6042 krb5kdc[17111](info): commencing operation
krb5kdc: starting...
W20260925 11:07:24.900080 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:25 dist-test-slave-6042 krb5kdc[17111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:07:25.155459 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39117
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:39117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:25.504937 17124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:25.505271 17124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:25.505419 17124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:25.513487 17124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:25.513622 17124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:25.513702 17124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:25.513788 17124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:25.513888 17124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:25.525319 17124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:39117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:25.526834 17124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:25.528348 17124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:25.541101 17129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:25.541172 17130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:25.541188 17132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:25.542680 17124 server_base.cc:1034] running on GCE node
I20260925 11:07:25.543577 17124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:25.545349 17124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:25.546593 17124 hybrid_clock.cc:648] HybridClock initialized: now 1790334445546543 us; error 54 us; skew 500 ppm
I20260925 11:07:25.547039 17124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T11:07:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1847 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:25 dist-test-slave-6042 krb5kdc[17138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:25 dist-test-slave-6042 krb5kdc[17138](info): set up 2 sockets
Sep 25 11:07:25 dist-test-slave-6042 krb5kdc[17138](info): commencing operation
krb5kdc: starting...
W20260925 11:07:26.779987 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:26 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:07:27.040210 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43081
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:27.381469 17151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:27.381820 17151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:27.381983 17151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:27.389550 17151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:27.389683 17151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:27.389731 17151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:27.389853 17151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:27.389914 17151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:27.400203 17151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:27.401674 17151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:27.403132 17151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:27.416093 17156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:27.416122 17159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:27.416044 17157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:27.417045 17151 server_base.cc:1034] running on GCE node
I20260925 11:07:27.417968 17151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:27.419612 17151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:27.420869 17151 hybrid_clock.cc:648] HybridClock initialized: now 1790334447420800 us; error 48 us; skew 500 ppm
I20260925 11:07:27.421329 17151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:27 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334447, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:27.425304 17151 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:07:27.427335 17151 webserver.cc:460] Webserver started at http://127.8.181.62:33341/ using document root <none> and password file <none>
I20260925 11:07:27.428208 17151 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:27.428360 17151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:27.428750 17151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:27.432430 17151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1"
format_stamp: "Formatted at 2026-09-25 11:07:27 on dist-test-slave-6042"
server_key: "fbe16c2e528ade5b66fba7c0a1c85747"
server_key_iv: "af28a8434b43b6b5df9565cb79246159"
server_key_version: "encryptionkey@0"
I20260925 11:07:27.433427 17151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1"
format_stamp: "Formatted at 2026-09-25 11:07:27 on dist-test-slave-6042"
server_key: "fbe16c2e528ade5b66fba7c0a1c85747"
server_key_iv: "af28a8434b43b6b5df9565cb79246159"
server_key_version: "encryptionkey@0"
I20260925 11:07:27.439851 17151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 11:07:27.444183 17166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:27.445801 17151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:07:27.446046 17151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1"
format_stamp: "Formatted at 2026-09-25 11:07:27 on dist-test-slave-6042"
server_key: "fbe16c2e528ade5b66fba7c0a1c85747"
server_key_iv: "af28a8434b43b6b5df9565cb79246159"
server_key_version: "encryptionkey@0"
I20260925 11:07:27.446296 17151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:27.471532 17151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:27.477067 17151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:27.477430 17151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:27.499316 17151 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:43081
I20260925 11:07:27.499337 17217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:43081 every 8 connection(s)
I20260925 11:07:27.500927 17151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:27.505330 17218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:27.506031 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17151
I20260925 11:07:27.506250 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:27.507246 8916 external_mini_cluster.cc:1428] Setting key d1cb460478a0f4714cd18dea8be27d6d
I20260925 11:07:27.521582 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Bootstrap starting.
Sep 25 11:07:27 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:27.525775 17218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:27.527117 17218 log.cc:824] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:27.530331 17218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: No bootstrap required, opened a new log
I20260925 11:07:27.534175 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:27.509676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.8.181.62:43081)
0925 11:07:27.510606 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:27.510618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:27.510663 (+ 45us) server_negotiation.cc:407] Connection header received
0925 11:07:27.512704 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:27.512731 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:27.513234 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:27.513608 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:27.515648 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:27.516979 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:27.519033 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:27.519803 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:27.519949 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:27.525699 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:27.525718 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:27.525737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:27.525790 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:27.529399 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:27.530310 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:27.530320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:27.530327 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:27.530661 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:27.531077 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:27.531082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:27.531086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:27.531663 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:27.531993 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:27.532546 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 11:07:27.533032 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":392,"threads_started":1}
I20260925 11:07:27.539148 17218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:27.539646 17218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:27.539790 17218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a9c3a1864e4ed8ad2a858503a8a8b1, State: Initialized, Role: FOLLOWER
I20260925 11:07:27.540434 17218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:27.540726 17218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:27.540889 17218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:27.541080 17218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:27.543937 17218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:27.544560 17218 leader_election.cc:302] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a9c3a1864e4ed8ad2a858503a8a8b1; no voters:
I20260925 11:07:27.545295 17218 leader_election.cc:288] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:27.545450 17223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:27.547186 17223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 LEADER]: Becoming Leader. State: Replica: 75a9c3a1864e4ed8ad2a858503a8a8b1, State: Running, Role: LEADER
I20260925 11:07:27.547788 17223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:27.548127 17218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:27.551429 17224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } } }
I20260925 11:07:27.551451 17225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a9c3a1864e4ed8ad2a858503a8a8b1. Latest consensus state: current_term: 1 leader_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } } }
I20260925 11:07:27.552219 17224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:27.552331 17225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:27.553845 17232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:27.559783 17232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:27.570704 17232 catalog_manager.cc:1347] Generated new cluster ID: 9d6d42f68c2d4c2792c4bc8620c6aff2
I20260925 11:07:27.570840 17232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:27.588120 17232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:27.589521 17232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:27.600008 17232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Generated new TSK 0
I20260925 11:07:27.600961 17232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:07:27.673303 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:28.041078 17244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:28.041455 17244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:28.041656 17244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:28.049863 17244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:28.050042 17244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:28.050271 17244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:28.061023 17244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:28.062614 17244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:28.064172 17244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:28.078693 17249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:28.078768 17252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:28.078804 17250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:28.079770 17244 server_base.cc:1034] running on GCE node
I20260925 11:07:28.080503 17244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:28.081737 17244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:28.082996 17244 hybrid_clock.cc:648] HybridClock initialized: now 1790334448082952 us; error 66 us; skew 500 ppm
I20260925 11:07:28.083419 17244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:28 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:28.094523 17244 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:07:28.096292 17244 webserver.cc:460] Webserver started at http://127.8.181.1:45021/ using document root <none> and password file <none>
I20260925 11:07:28.097083 17244 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:28.097268 17244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:28.097606 17244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:28.101143 17244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "d714c1299c7540c3be67442ff8d30a22"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "2a9ad093ef544200cf02eb2d3d2d378c"
server_key_iv: "1504eece6f365345742b2a4cfff34424"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.102015 17244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "d714c1299c7540c3be67442ff8d30a22"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "2a9ad093ef544200cf02eb2d3d2d378c"
server_key_iv: "1504eece6f365345742b2a4cfff34424"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.107929 17244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 11:07:28.112066 17259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.113852 17244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260925 11:07:28.114087 17244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "d714c1299c7540c3be67442ff8d30a22"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "2a9ad093ef544200cf02eb2d3d2d378c"
server_key_iv: "1504eece6f365345742b2a4cfff34424"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.114339 17244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:28.128820 17244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:28.133433 17244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:28.133769 17244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:28.134898 17244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:28.136351 17244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:28.136461 17244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.136605 17244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:28.136689 17244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.174185 17244 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33971
I20260925 11:07:28.174281 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33971 every 8 connection(s)
I20260925 11:07:28.175583 17244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:28.183161 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17244
I20260925 11:07:28.183383 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:28.184098 8916 external_mini_cluster.cc:1428] Setting key 00b0fab9c57e682ae528c10717071da6
Sep 25 11:07:28 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:28.197291 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:28.178194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:43553 (local address 127.8.181.62:43081)
0925 11:07:28.178819 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:28.178823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:28.179800 (+ 977us) server_negotiation.cc:407] Connection header received
0925 11:07:28.181774 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:28.181780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:28.181903 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:28.182030 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:28.183906 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.184739 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.186776 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.187184 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.187326 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:28.191000 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:28.191006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:28.191009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:28.191046 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:28.193827 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:28.194796 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:28.194801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:28.194803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:28.194932 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:28.195417 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:28.195421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:28.195423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:28.195715 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:28.195874 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:28.196667 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 11:07:28.196918 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":397,"threads_started":1}
I20260925 11:07:28.198331 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:28.178832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:43553)
0925 11:07:28.179608 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:28.179654 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:28.181445 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:28.182302 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:28.182311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:28.182939 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:28.183702 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:28.183725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.184916 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.184920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:28.186623 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:28.186638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.187431 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.187440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:28.187701 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:28.188433 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:28.188457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:28.190728 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:28.194032 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:28.194043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:28.194071 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:28.194636 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:28.195056 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:28.195061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:28.195065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:28.195283 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:28.195859 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:28.195867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:28.196253 (+ 386us) client_negotiation.cc:769] Sending connection context
0925 11:07:28.196529 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 11:07:28.197116 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":314,"threads_started":1}
I20260925 11:07:28.200318 17372 heartbeater.cc:347] Connected to a master server at 127.8.181.62:43081
I20260925 11:07:28.200665 17372 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:28.201478 17372 heartbeater.cc:511] Master 127.8.181.62:43081 requested a full tablet report, sending...
I20260925 11:07:28.203964 17183 ts_manager.cc:194] Registered new tserver with Master: d714c1299c7540c3be67442ff8d30a22 (127.8.181.1:33971)
I20260925 11:07:28.205514 17183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:43553
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:07:28.243132 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:28.603601 17378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:28.603943 17378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:28.604090 17378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:28.612376 17378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:28.612512 17378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:28.612716 17378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:28.623746 17378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:28.625411 17378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:28.626945 17378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:28.642899 17383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:28.642710 17384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:28.642678 17386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:28.643749 17378 server_base.cc:1034] running on GCE node
I20260925 11:07:28.644465 17378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:28.645561 17378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:28.646801 17378 hybrid_clock.cc:648] HybridClock initialized: now 1790334448646754 us; error 51 us; skew 500 ppm
I20260925 11:07:28.647217 17378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:28 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:28.651480 17378 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:07:28.653434 17378 webserver.cc:460] Webserver started at http://127.8.181.2:35643/ using document root <none> and password file <none>
I20260925 11:07:28.654227 17378 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:28.654388 17378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:28.654755 17378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:28.657975 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "6141d4a0d150462e9948143b2049c6ba"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "8b2d5cdb1fdf57ced1bee02bd6b7dce8"
server_key_iv: "2a2c2c3af8d97a48595896100756a40e"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.658926 17378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "6141d4a0d150462e9948143b2049c6ba"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "8b2d5cdb1fdf57ced1bee02bd6b7dce8"
server_key_iv: "2a2c2c3af8d97a48595896100756a40e"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.664862 17378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 11:07:28.669173 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.671046 17378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260925 11:07:28.671267 17378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "6141d4a0d150462e9948143b2049c6ba"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "8b2d5cdb1fdf57ced1bee02bd6b7dce8"
server_key_iv: "2a2c2c3af8d97a48595896100756a40e"
server_key_version: "encryptionkey@0"
I20260925 11:07:28.671494 17378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:28.713163 17378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:28.718600 17378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:28.718950 17378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:28.720445 17378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:28.721995 17378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:28.722077 17378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.722235 17378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:28.722294 17378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:28.760068 17378 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44303
I20260925 11:07:28.760180 17505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44303 every 8 connection(s)
I20260925 11:07:28.761482 17378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:28.770597 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17378
I20260925 11:07:28.770814 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:28.771327 8916 external_mini_cluster.cc:1428] Setting key a10776f135f57de4fb94ca01fc9df6c2
Sep 25 11:07:28 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334448, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:28.784448 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:28.764389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42329 (local address 127.8.181.62:43081)
0925 11:07:28.764921 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:28.764926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:28.765721 (+ 795us) server_negotiation.cc:407] Connection header received
0925 11:07:28.767755 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:28.767763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:28.767878 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:28.768009 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:28.769631 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.770412 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.772437 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.772980 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.773152 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:28.777473 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:28.777479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:28.777482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:28.777525 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:28.780674 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:28.781608 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:28.781620 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:28.781622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:28.781764 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:28.782272 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:28.782284 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:28.782286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:28.782712 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:28.782895 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:28.783740 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 11:07:28.784023 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":306,"threads_started":1}
I20260925 11:07:28.785645 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:28.764760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:42329)
0925 11:07:28.765496 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:28.765536 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:28.767412 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:28.768277 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:28.768287 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:28.768838 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:28.769430 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:28.769449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.770633 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.770639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:28.772285 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:28.772299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:28.773254 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:28.773262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:28.773522 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:28.774395 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:28.774421 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:28.777177 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:28.780881 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:28.780893 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:28.780915 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:28.781445 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:28.781898 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:28.781903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:28.781907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:28.782129 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:28.782868 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:28.782877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:28.783279 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 11:07:28.783625 (+ 346us) client_negotiation.cc:240] Negotiation successful
0925 11:07:28.784287 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":310,"threads_started":1}
I20260925 11:07:28.787369 17506 heartbeater.cc:347] Connected to a master server at 127.8.181.62:43081
I20260925 11:07:28.787742 17506 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:28.788555 17506 heartbeater.cc:511] Master 127.8.181.62:43081 requested a full tablet report, sending...
I20260925 11:07:28.790231 17183 ts_manager.cc:194] Registered new tserver with Master: 6141d4a0d150462e9948143b2049c6ba (127.8.181.2:44303)
I20260925 11:07:28.791265 17183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:42329
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:07:28.827263 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:29.178758 17512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:29.179124 17512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:29.179337 17512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:29.187973 17512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:29.188119 17512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:29.188324 17512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:29.199877 17512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:29.201387 17512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:29.202955 17512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 11:07:29.209322 17372 heartbeater.cc:503] Master 127.8.181.62:43081 was elected leader, sending a full tablet report...
W20260925 11:07:29.218583 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:29.218446 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:29.219080 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:29.219570 17512 server_base.cc:1034] running on GCE node
I20260925 11:07:29.220384 17512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:29.221632 17512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:29.222888 17512 hybrid_clock.cc:648] HybridClock initialized: now 1790334449222842 us; error 69 us; skew 500 ppm
I20260925 11:07:29.223289 17512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:29 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334449, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:29.227370 17512 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:07:29.229153 17512 webserver.cc:460] Webserver started at http://127.8.181.3:35899/ using document root <none> and password file <none>
I20260925 11:07:29.229974 17512 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:29.230129 17512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:29.230430 17512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:29.233590 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "952e4af6a432413ca5d278049417b3e9"
format_stamp: "Formatted at 2026-09-25 11:07:29 on dist-test-slave-6042"
server_key: "9a73bd071b7d5a834cf9d7f0ffc3a401"
server_key_iv: "89cfdb918fe348c8b933e8d6aa9aefca"
server_key_version: "encryptionkey@0"
I20260925 11:07:29.234522 17512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "952e4af6a432413ca5d278049417b3e9"
format_stamp: "Formatted at 2026-09-25 11:07:29 on dist-test-slave-6042"
server_key: "9a73bd071b7d5a834cf9d7f0ffc3a401"
server_key_iv: "89cfdb918fe348c8b933e8d6aa9aefca"
server_key_version: "encryptionkey@0"
I20260925 11:07:29.240438 17512 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 11:07:29.244623 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:29.246456 17512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:07:29.246789 17512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "952e4af6a432413ca5d278049417b3e9"
format_stamp: "Formatted at 2026-09-25 11:07:29 on dist-test-slave-6042"
server_key: "9a73bd071b7d5a834cf9d7f0ffc3a401"
server_key_iv: "89cfdb918fe348c8b933e8d6aa9aefca"
server_key_version: "encryptionkey@0"
I20260925 11:07:29.247004 17512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:29.274415 17512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:29.279711 17512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:29.279983 17512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:29.280957 17512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:29.282374 17512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:29.282446 17512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:29.282596 17512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:29.282645 17512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:29.320883 17512 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:43205
I20260925 11:07:29.320945 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:43205 every 8 connection(s)
I20260925 11:07:29.322450 17512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:29.332468 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17512
I20260925 11:07:29.332628 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:29.333159 8916 external_mini_cluster.cc:1428] Setting key b059972d315770a966d3fddad5e98e2b
Sep 25 11:07:29 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334449, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:29.345016 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:29.325482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55147 (local address 127.8.181.62:43081)
0925 11:07:29.326119 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:29.326126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:29.327261 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 11:07:29.329136 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:29.329140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:29.329233 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:29.329376 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:29.331269 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.332057 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.333886 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.334440 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.334604 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:29.338460 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:29.338467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:29.338599 (+ 132us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:29.338645 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:29.341601 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.342464 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.342533 (+ 69us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.342537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.342742 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.343185 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.343188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.343190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.343457 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:29.343622 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:29.344443 (+ 821us) server_negotiation.cc:299] Negotiation successful
0925 11:07:29.344724 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":294,"threads_started":1}
I20260925 11:07:29.345927 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:29.326145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:55147)
0925 11:07:29.327086 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:29.327125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:29.328826 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:29.329620 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:29.329631 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:29.330222 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:29.331017 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:29.331036 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.332252 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.332257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:29.333724 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:29.333737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.334709 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.334718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:29.334974 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:29.335718 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:29.335751 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:29.338201 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:29.341778 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:29.341787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:29.341802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:29.342322 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:29.342869 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:29.342873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:29.342875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:29.343077 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:29.343599 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:29.343606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:29.344031 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 11:07:29.344310 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 11:07:29.344773 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":444,"threads_started":1}
I20260925 11:07:29.347697 17640 heartbeater.cc:347] Connected to a master server at 127.8.181.62:43081
I20260925 11:07:29.348067 17640 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:29.348877 17640 heartbeater.cc:511] Master 127.8.181.62:43081 requested a full tablet report, sending...
I20260925 11:07:29.350541 17183 ts_manager.cc:194] Registered new tserver with Master: 952e4af6a432413ca5d278049417b3e9 (127.8.181.3:43205)
I20260925 11:07:29.351469 17183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:55147
I20260925 11:07:29.362386 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
Entry for principal oryx/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab.
WARNING: no policy specified for oryx/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
Entry for principal oryx/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab.
WARNING: no policy specified for oryx/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
Entry for principal oryx/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab.
I20260925 11:07:29.522388 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17244
Sep 25 11:07:29 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:07:29.541666 17651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:29.523700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.8.181.1:33971)
0925 11:07:29.524627 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:29.524634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:29.524662 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:07:29.524723 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:29.524729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:29.524948 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:29.525105 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:29.526734 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.527793 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.529224 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.529972 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.530106 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:29.534753 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:29.534767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:29.534783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:29.534829 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:29.538273 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.539182 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.539190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.539192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.539338 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.539892 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.539897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.539899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.540309 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:29.540557 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:29.540964 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 11:07:29.541211 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":599,"threads_started":1}
I20260925 11:07:29.543175 17367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:29.628492 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17244
I20260925 11:07:29.645720 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17378
Sep 25 11:07:29 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:07:29.664448 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:29.646964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.8.181.2:44303)
0925 11:07:29.647561 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:29.647571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:29.647600 (+ 29us) server_negotiation.cc:407] Connection header received
0925 11:07:29.647665 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:29.647672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:29.647987 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:29.648185 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:29.649509 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.650754 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.652354 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.653125 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.653292 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:29.657985 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:29.658003 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:29.658027 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:29.658076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:29.661226 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.662042 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.662049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.662052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.662177 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.662854 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.662860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.662862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.663266 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:29.663528 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:29.663883 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 11:07:29.664135 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":296,"threads_started":1}
I20260925 11:07:29.666060 17501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:29.753710 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17378
I20260925 11:07:29.770387 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17512
Sep 25 11:07:29 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:07:29.789867 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:29.771871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.8.181.3:43205)
0925 11:07:29.772529 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:29.772537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:29.772585 (+ 48us) server_negotiation.cc:407] Connection header received
0925 11:07:29.772646 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:29.772653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:29.772882 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:29.773020 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:29.774783 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.775839 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.777254 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:29.777919 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:29.778040 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:29.782976 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:29.782993 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:29.783010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:29.783099 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:29.786422 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.787425 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.787433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.787436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.787603 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:29.788106 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:29.788110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:29.788113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:29.788516 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:29.788829 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:29.789193 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 11:07:29.789448 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":361,"threads_started":1}
I20260925 11:07:29.791460 17635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:29.876135 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17512
I20260925 11:07:29.891786 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17151
I20260925 11:07:29.893118 17213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:29.998730 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17151
I20260925 11:07:30.019876 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43081
--webserver_interface=127.8.181.62
--webserver_port=33341
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:43081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:30.366582 17658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:30.366976 17658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:30.367136 17658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:30.375171 17658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:30.375348 17658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:30.375463 17658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:30.375591 17658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:30.375674 17658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:30.387544 17658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:43081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:30.389127 17658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:30.390762 17658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:30.404333 17664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:30.404358 17663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:30.404407 17666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:30.405278 17658 server_base.cc:1034] running on GCE node
I20260925 11:07:30.406143 17658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:30.407816 17658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:30.409112 17658 hybrid_clock.cc:648] HybridClock initialized: now 1790334450409010 us; error 63 us; skew 500 ppm
I20260925 11:07:30.409508 17658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:30 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334450, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:30.412911 17658 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:07:30.414791 17658 webserver.cc:460] Webserver started at http://127.8.181.62:33341/ using document root <none> and password file <none>
I20260925 11:07:30.415576 17658 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:30.415747 17658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:30.422108 17658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 11:07:30.425915 17673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:30.427994 17658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:30.428229 17658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1"
format_stamp: "Formatted at 2026-09-25 11:07:27 on dist-test-slave-6042"
server_key: "fbe16c2e528ade5b66fba7c0a1c85747"
server_key_iv: "af28a8434b43b6b5df9565cb79246159"
server_key_version: "encryptionkey@0"
I20260925 11:07:30.429358 17658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:30.447456 17658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:30.452550 17658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:30.452874 17658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:30.474884 17658 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:43081
I20260925 11:07:30.474905 17724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:43081 every 8 connection(s)
I20260925 11:07:30.476734 17658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:30.484726 17725 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:07:30.485289 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17658
I20260925 11:07:30.486150 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:33971
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:43081
--webserver_port=45021
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:07:30.488520 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Bootstrap starting.
I20260925 11:07:30.515563 17725 log.cc:824] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:30.528172 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:07:30.528867 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Bootstrap complete.
I20260925 11:07:30.542850 17725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:30.544276 17725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a9c3a1864e4ed8ad2a858503a8a8b1, State: Initialized, Role: FOLLOWER
I20260925 11:07:30.545122 17725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:30.545432 17725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:30.545562 17725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:30.545775 17725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:07:30.551988 17725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:30.552707 17725 leader_election.cc:302] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a9c3a1864e4ed8ad2a858503a8a8b1; no voters:
I20260925 11:07:30.553426 17725 leader_election.cc:288] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 11:07:30.553627 17729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 11:07:30.555096 17729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [term 2 LEADER]: Becoming Leader. State: Replica: 75a9c3a1864e4ed8ad2a858503a8a8b1, State: Running, Role: LEADER
I20260925 11:07:30.555835 17729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } }
I20260925 11:07:30.556368 17725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:30.558840 17731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a9c3a1864e4ed8ad2a858503a8a8b1. Latest consensus state: current_term: 2 leader_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } } }
I20260925 11:07:30.559273 17731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:30.559798 17730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a9c3a1864e4ed8ad2a858503a8a8b1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43081 } } }
I20260925 11:07:30.560165 17730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:30.562676 17734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:30.569504 17734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:30.571367 17734 catalog_manager.cc:1259] Loaded cluster ID: 9d6d42f68c2d4c2792c4bc8620c6aff2
I20260925 11:07:30.571581 17734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:30.576393 17734 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:30.578457 17734 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75a9c3a1864e4ed8ad2a858503a8a8b1: Loaded TSK: 0
I20260925 11:07:30.580578 17734 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 11:07:30.879093 17727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:30.879454 17727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:30.879670 17727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:30.887605 17727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:30.887735 17727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:30.887914 17727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:30.898751 17727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.1.keytab
--principal=oryx/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:33971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=45021
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:30.900259 17727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:30.901669 17727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:30.916549 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:30.916545 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:30.917402 17754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:30.917514 17727 server_base.cc:1034] running on GCE node
I20260925 11:07:30.918236 17727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:30.919579 17727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:30.920857 17727 hybrid_clock.cc:648] HybridClock initialized: now 1790334450920775 us; error 63 us; skew 500 ppm
I20260925 11:07:30.921253 17727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:30 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334450, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:30.924924 17727 init.cc:377] Logged in from keytab as oryx/127.8.181.1@KRBTEST.COM (short username oryx)
I20260925 11:07:30.926848 17727 webserver.cc:460] Webserver started at http://127.8.181.1:45021/ using document root <none> and password file <none>
I20260925 11:07:30.927605 17727 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:30.927770 17727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:30.935148 17727 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 11:07:30.938558 17761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:30.940260 17727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:30.940478 17727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "d714c1299c7540c3be67442ff8d30a22"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "2a9ad093ef544200cf02eb2d3d2d378c"
server_key_iv: "1504eece6f365345742b2a4cfff34424"
server_key_version: "encryptionkey@0"
I20260925 11:07:30.941525 17727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:30.961710 17727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:30.968719 17727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:30.969107 17727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:30.970566 17727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:30.972506 17727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:30.972632 17727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:30.972764 17727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:30.972852 17727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.012501 17727 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33971
I20260925 11:07:31.012660 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33971 every 8 connection(s)
I20260925 11:07:31.014171 17727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:31.016830 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17727
I20260925 11:07:31.017584 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:44303
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:43081
--webserver_port=35643
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:31.039119 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.018178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:51797)
0925 11:07:31.019263 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.019318 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.022134 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.024178 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.024193 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.025043 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.025914 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.025940 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.028834 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.028840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.030332 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.030349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.032049 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.032056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.032284 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.033225 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.033251 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.037982 (+ 4731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":550,"threads_started":1}
W20260925 11:07:31.039888 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.018003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:51797 (local address 127.8.181.62:43081)
0925 11:07:31.018914 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.018932 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.019488 (+ 556us) server_negotiation.cc:407] Connection header received
0925 11:07:31.023075 (+ 3587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.023092 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.023527 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.023899 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.026186 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.028591 (+ 2405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.030582 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.031775 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.032000 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.038586 (+ 6586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:51797: BlockingRecv error: recv got EOF from 127.8.181.1:51797 (error 108)
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":278,"threads_started":1}
W20260925 11:07:31.039911 17874 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:43081 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.181.62:43081: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:31.049183 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.040606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:45695)
0925 11:07:31.040808 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.040835 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.040935 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.041307 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.041311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.041629 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.041792 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.041801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.042963 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.042970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.043925 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.043937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.044659 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.044665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.044872 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.045553 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.045570 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.048856 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 11:07:31.049775 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.040709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:45695 (local address 127.8.181.62:43081)
0925 11:07:31.040907 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.040915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.040940 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:07:31.041049 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.041058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.041154 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.041279 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.041963 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.042812 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.044087 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.044499 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.044643 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.049107 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:45695: BlockingRecv error: recv got EOF from 127.8.181.1:45695 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:31.059478 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.049961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:60137)
0925 11:07:31.050135 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.050162 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.050251 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.050915 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.050920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.051228 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.051405 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.051414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.052925 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.052929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.054027 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.054041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.055040 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.055045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.055212 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.055862 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.055881 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.059251 (+ 3370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 11:07:31.059820 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.050206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:60137 (local address 127.8.181.62:43081)
0925 11:07:31.050545 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.050553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.050575 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:07:31.050642 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.050646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.050749 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.050869 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.051553 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.052719 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.054192 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.054880 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.055021 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.059566 (+ 4545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:60137: BlockingRecv error: recv got EOF from 127.8.181.1:60137 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
W20260925 11:07:31.059846 17874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:07:31.379626 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:31.379935 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:31.380029 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:31.387501 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:31.387655 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:31.387773 17878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:31.397998 17878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.2.keytab
--principal=oryx/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:44303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=35643
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:31.399449 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:31.400815 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:31.415549 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:31.415699 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:31.416105 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:31.417425 17878 server_base.cc:1034] running on GCE node
I20260925 11:07:31.418215 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:31.419445 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:31.420698 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790334451420627 us; error 66 us; skew 500 ppm
I20260925 11:07:31.421059 17878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334451, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:31.424698 17878 init.cc:377] Logged in from keytab as oryx/127.8.181.2@KRBTEST.COM (short username oryx)
I20260925 11:07:31.426560 17878 webserver.cc:460] Webserver started at http://127.8.181.2:35643/ using document root <none> and password file <none>
I20260925 11:07:31.427551 17878 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:31.427683 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:31.434163 17878 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 11:07:31.437883 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.439845 17878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:07:31.440057 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "6141d4a0d150462e9948143b2049c6ba"
format_stamp: "Formatted at 2026-09-25 11:07:28 on dist-test-slave-6042"
server_key: "8b2d5cdb1fdf57ced1bee02bd6b7dce8"
server_key_iv: "2a2c2c3af8d97a48595896100756a40e"
server_key_version: "encryptionkey@0"
I20260925 11:07:31.441083 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:31.461095 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:31.466403 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:31.466871 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:31.468106 17878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:31.469689 17878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:31.469779 17878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.469919 17878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:31.470006 17878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.515829 17878 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44303
I20260925 11:07:31.515930 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44303 every 8 connection(s)
I20260925 11:07:31.517501 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:31.518055 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 17878
I20260925 11:07:31.519084 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:43205
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:43081
--webserver_port=35899
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:31.539371 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.521360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:48635 (local address 127.8.181.62:43081)
0925 11:07:31.521589 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.521596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.523173 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 11:07:31.525131 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.525141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.525253 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.525387 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.527729 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.528459 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.531339 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.531889 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.532016 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.539012 (+ 6996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:48635: BlockingRecv error: recv got EOF from 127.8.181.2:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:07:31.539875 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.521917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:48635)
0925 11:07:31.522958 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.523018 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.524794 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.525617 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.525626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.526531 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.527418 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.527442 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.528683 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.528692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.531106 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.531130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.532182 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.532192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.532551 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.533383 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.533447 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.538669 (+ 5222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":429,"threads_started":1}
W20260925 11:07:31.540668 18006 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:43081 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.181.62:43081: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:31.552251 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.541375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:55505)
0925 11:07:31.541567 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.541596 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.541742 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.542098 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.542103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.542541 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.543145 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.543165 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.544477 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.544485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.545974 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.545994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.547195 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.547202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.547422 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.548346 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.548381 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.551920 (+ 3539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 11:07:31.552946 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.541595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:55505 (local address 127.8.181.62:43081)
0925 11:07:31.541793 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.541801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.541822 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:31.541888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.541893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.541962 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.542070 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.543367 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.544302 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.546167 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.547001 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.547174 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.552605 (+ 5431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:55505: BlockingRecv error: recv got EOF from 127.8.181.2:55505 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:31.564910 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:31.553238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:52789)
0925 11:07:31.553755 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:31.553781 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:31.553941 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:31.554688 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:31.554694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:31.555080 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:31.555259 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.555276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.556532 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.556541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.558161 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:31.558188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.559450 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.559463 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:31.559778 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.560785 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:31.560812 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:31.564538 (+ 3726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328}
W20260925 11:07:31.565299 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:31.553559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:52789 (local address 127.8.181.62:43081)
0925 11:07:31.553808 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:31.553817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:31.553847 (+ 30us) server_negotiation.cc:407] Connection header received
0925 11:07:31.554300 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:31.554305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:31.554399 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:31.554566 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:31.555450 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.556369 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.558426 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:31.559096 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:31.559445 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:31.564845 (+ 5400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:52789: BlockingRecv error: recv got EOF from 127.8.181.2:52789 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260925 11:07:31.565425 18006 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:07:31.888214 18009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:31.888528 18009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:31.888630 18009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:31.896077 18009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:31.896219 18009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:31.896447 18009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:31.907001 18009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/krb5kdc/oryx_127.8.181.3.keytab
--principal=oryx/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:43205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=35899
--tserver_master_addrs=127.8.181.62:43081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:31.908427 18009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:31.909803 18009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:31.925539 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:31.925539 18017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:31.925995 18014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:31.927063 18009 server_base.cc:1034] running on GCE node
I20260925 11:07:31.928085 18009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:31.929234 18009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:31.930637 18009 hybrid_clock.cc:648] HybridClock initialized: now 1790334451930564 us; error 54 us; skew 500 ppm
I20260925 11:07:31.931120 18009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:31 dist-test-slave-6042 krb5kdc[17138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334451, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:31.935213 18009 init.cc:377] Logged in from keytab as oryx/127.8.181.3@KRBTEST.COM (short username oryx)
I20260925 11:07:31.937448 18009 webserver.cc:460] Webserver started at http://127.8.181.3:35899/ using document root <none> and password file <none>
I20260925 11:07:31.938237 18009 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:31.938414 18009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:31.944706 18009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:07:31.948576 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.950258 18009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:07:31.950589 18009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "952e4af6a432413ca5d278049417b3e9"
format_stamp: "Formatted at 2026-09-25 11:07:29 on dist-test-slave-6042"
server_key: "9a73bd071b7d5a834cf9d7f0ffc3a401"
server_key_iv: "89cfdb918fe348c8b933e8d6aa9aefca"
server_key_version: "encryptionkey@0"
I20260925 11:07:31.951710 18009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:31.981348 18009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:31.986857 18009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:31.987257 18009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:31.988509 18009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:31.990191 18009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:31.990336 18009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:31.990556 18009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:31.990666 18009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 11:07:32.030260 18009 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:43205
I20260925 11:07:32.030323 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:43205 every 8 connection(s)
I20260925 11:07:32.032016 18009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:32.039891 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18009
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:32.054808 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:32.035372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:45069 (local address 127.8.181.62:43081)
0925 11:07:32.035585 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.035589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.037137 (+ 1548us) server_negotiation.cc:407] Connection header received
0925 11:07:32.039268 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.039274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.039379 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.039506 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.041870 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.043904 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.046908 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.047398 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.047498 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.054448 (+ 6950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:45069: BlockingRecv error: recv got EOF from 127.8.181.3:45069 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:07:32.055352 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.035965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:45069)
0925 11:07:32.036873 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:32.036931 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:32.038897 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:32.039799 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:32.039809 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:32.040495 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:32.041576 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.041608 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.044221 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.044229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.046717 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.046736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.047679 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.047687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.048125 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.049271 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:32.049296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:32.054049 (+ 4753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":351,"threads_started":1}
W20260925 11:07:32.056277 18137 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:43081 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.181.62:43081: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:32.065901 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.041177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.8.181.62:43081)
0925 11:07:32.042393 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.042401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.042438 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:07:32.042901 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.042907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.043017 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.043181 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.045857 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.046958 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.048896 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.049441 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.049570 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.053021 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:32.053036 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:32.053057 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:32.053107 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:32.057691 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:32.058627 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:32.058637 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:32.058648 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:32.058847 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:32.059288 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:32.059293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:32.059297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:32.059944 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:32.060257 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:32.064476 (+ 4219us) server_negotiation.cc:299] Negotiation successful
0925 11:07:32.065439 (+ 963us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":643,"threads_started":1}
I20260925 11:07:32.066416 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.057190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:46759)
0925 11:07:32.057400 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:32.057422 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:32.057525 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:32.057967 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:32.057972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:32.058397 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:32.058615 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.058624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.059680 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.059686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.060974 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.060990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.061808 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.061814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.061978 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.062648 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:32.062666 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:32.065813 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 11:07:32.066820 17876 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:32.057514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:46759 (local address 127.8.181.62:43081)
0925 11:07:32.057706 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.057710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.057731 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:32.057776 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.057779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.057831 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.057940 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.058764 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.059540 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.061156 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.061642 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.061783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.066587 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:46759: BlockingRecv error: recv got EOF from 127.8.181.3:46759 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:32.077031 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.060438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:54031)
0925 11:07:32.067368 (+ 6930us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:32.067388 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:32.067616 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:32.068276 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:32.068281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:32.068780 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:32.069058 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.069072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.070574 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.070580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.071637 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.071651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.072471 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.072477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.072664 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.073324 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:32.073344 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:32.076708 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6777,"thread_start_us":6924,"threads_started":1}
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:32.077461 18141 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:32.060507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:54031 (local address 127.8.181.62:43081)
0925 11:07:32.066248 (+ 5741us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.066255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.067497 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 11:07:32.067846 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.067855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.067941 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.068022 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.069270 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.070361 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.071806 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.072268 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.072409 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.077111 (+ 4702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:54031: BlockingRecv error: recv got EOF from 127.8.181.1:54031 (error 108)
Metrics: {"server-negotiator.queue_time_us":5589,"thread_start_us":8152,"threads_started":1}
I20260925 11:07:32.078312 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.067739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:57909)
0925 11:07:32.068056 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:32.068076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:32.068174 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:32.069602 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:32.069608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:32.070058 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:32.070261 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.070274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.071711 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.071717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.072702 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.072714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.073467 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.073472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.073686 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.074376 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:32.074422 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:32.078090 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203}
W20260925 11:07:32.078783 18137 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:07:32.079181 18143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:32.068856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57909 (local address 127.8.181.62:43081)
0925 11:07:32.069250 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.069257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.069275 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:07:32.069330 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.069335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.069411 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.069549 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.070457 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.071527 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.072874 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.073309 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.073446 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.078443 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:57909: BlockingRecv error: recv got EOF from 127.8.181.3:57909 (error 108)
Metrics: {"server-negotiator.queue_time_us":263}
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:32 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:32.576699 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:32.566371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:41077)
0925 11:07:32.567027 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:32.567055 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:32.567240 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:32.567792 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:32.567797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:32.568436 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:32.568654 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.568664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.569872 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.569877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.571005 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:32.571020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.571964 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.571970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:32.572118 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.572891 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:32.572912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:32.576412 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":380,"threads_started":1}
W20260925 11:07:32.577138 18143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:32.566531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:41077 (local address 127.8.181.62:43081)
0925 11:07:32.566736 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:32.566744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:32.567165 (+ 421us) server_negotiation.cc:407] Connection header received
0925 11:07:32.567392 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:32.567395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:32.567467 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:32.567552 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:32.568821 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.569711 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.571187 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:32.571718 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:32.571847 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:32.576786 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:41077: BlockingRecv error: recv got EOF from 127.8.181.2:41077 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:33.091456 18146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:33.078555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:37453 (local address 127.8.181.62:43081)
0925 11:07:33.079121 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:33.079130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:33.079463 (+ 333us) server_negotiation.cc:407] Connection header received
0925 11:07:33.079677 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:33.079685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:33.079797 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:33.079910 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:33.081091 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.082908 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.084467 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.085104 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.085202 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.091179 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:37453: BlockingRecv error: recv got EOF from 127.8.181.1:37453 (error 108)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":321,"threads_started":1}
I20260925 11:07:33.091513 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:33.078373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:37453)
0925 11:07:33.079342 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:33.079367 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:33.079494 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:33.079963 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:33.079971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:33.080455 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:33.080613 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.080623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.083079 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.083086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.084288 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.084304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.085316 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.085323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.085511 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.086421 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:33.086448 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:33.090811 (+ 4363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":703,"threads_started":1}
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:33.094238 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:33.079458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:42707)
0925 11:07:33.079985 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:33.080008 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:33.080122 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:33.082621 (+ 2499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:33.082629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:33.083285 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:33.083528 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.083542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.085066 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.085071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.086694 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.086715 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.087851 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.087859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.088015 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.088821 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:33.088850 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:33.093912 (+ 5062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
W20260925 11:07:33.095094 18148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:33.080872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:42707 (local address 127.8.181.62:43081)
0925 11:07:33.081745 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:33.081752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:33.081778 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:07:33.081847 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:33.081852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:33.082081 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:33.082630 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:33.083804 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.084845 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.086990 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.087564 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.087765 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.094444 (+ 6679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:42707: BlockingRecv error: recv got EOF from 127.8.181.3:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":384,"threads_started":1}
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:33 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:33.589890 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:33.578022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:41101)
0925 11:07:33.578707 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:33.578729 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:33.578933 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:33.579406 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:33.579411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:33.579889 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:33.580046 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.580056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.581169 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.581176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.582122 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:33.582141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.583028 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.583034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:33.583382 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.584600 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:33.584626 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:33.589614 (+ 4988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":390,"threads_started":1}
W20260925 11:07:33.590277 18148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:33.578308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:41101 (local address 127.8.181.62:43081)
0925 11:07:33.578618 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:33.578625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:33.578835 (+ 210us) server_negotiation.cc:407] Connection header received
0925 11:07:33.579145 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:33.579151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:33.579238 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:33.579374 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:33.580216 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.581008 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.582305 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:33.582832 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:33.582974 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:33.589965 (+ 6991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:41101: BlockingRecv error: recv got EOF from 127.8.181.2:41101 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:34.105273 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:34.093153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:38849)
0925 11:07:34.093857 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:34.093880 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:34.094037 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:34.095166 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:34.095176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:34.095736 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:34.096152 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.096180 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.097560 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.097567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.099149 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.099163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.100377 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.100383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.100593 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.101303 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:34.101327 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:34.104866 (+ 3539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":497,"threads_started":1}
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:34.105499 18151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:34.093483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:38849 (local address 127.8.181.62:43081)
0925 11:07:34.093964 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:34.093969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:34.093988 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:07:34.094385 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:34.094391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:34.094863 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:34.094993 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:34.096364 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.097378 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.099301 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.100046 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.100194 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.105178 (+ 4984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:38849: BlockingRecv error: recv got EOF from 127.8.181.1:38849 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":382,"thread_start_us":280,"threads_started":1}
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:34.107587 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:34.095585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:57667)
0925 11:07:34.096240 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:34.096273 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:34.096406 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:34.097630 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:34.097660 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:34.098216 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:34.098392 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.098404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.100355 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.100361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.101753 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.101774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.102816 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.102823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.103022 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.103743 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:34.103763 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:34.107355 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":421,"threads_started":1}
W20260925 11:07:34.107937 18153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:34.095734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57667 (local address 127.8.181.62:43081)
0925 11:07:34.097289 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:34.097296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:34.097313 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:34.097360 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:34.097365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:34.097440 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:34.097594 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:34.098749 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.100082 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.101934 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.102620 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.102796 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.107660 (+ 4864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:57667: BlockingRecv error: recv got EOF from 127.8.181.3:57667 (error 108)
Metrics: {"server-negotiator.queue_time_us":1412,"thread_start_us":1241,"threads_started":1}
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:34 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:34.602604 18154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:34.591472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:35317)
0925 11:07:34.592167 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:34.592206 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:34.592379 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:34.592949 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:34.592956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:34.593453 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:34.593723 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.593735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.595486 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.595494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.596648 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:34.596663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.597607 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.597613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:34.597766 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.598628 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:34.598649 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:34.602218 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":421,"threads_started":1}
W20260925 11:07:34.603031 18153 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:34.591617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35317 (local address 127.8.181.62:43081)
0925 11:07:34.592060 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:34.592070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:34.592366 (+ 296us) server_negotiation.cc:407] Connection header received
0925 11:07:34.592499 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:34.592504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:34.592617 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:34.592754 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:34.593889 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.595160 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.596891 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:34.597399 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:34.597523 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:34.602692 (+ 5169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:35317: BlockingRecv error: recv got EOF from 127.8.181.2:35317 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:35.118168 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:35.106618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:57065)
0925 11:07:35.107080 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:35.107098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:35.107283 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:35.107788 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:35.107793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:35.108337 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:35.108525 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.108539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.110969 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.110976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.112083 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.112114 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.113096 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.113103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.113309 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.114146 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:35.114166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:35.117904 (+ 3738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
W20260925 11:07:35.118652 18156 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:35.106741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:57065 (local address 127.8.181.62:43081)
0925 11:07:35.107431 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:35.107435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:35.107452 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:35.107512 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:35.107535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:35.107596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:35.107692 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:35.108852 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.110801 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.112284 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.112889 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.113074 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.118328 (+ 5254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:57065: BlockingRecv error: recv got EOF from 127.8.181.1:57065 (error 108)
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":570,"thread_start_us":470,"threads_started":1}
I20260925 11:07:35.118966 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:35.108804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:53367)
0925 11:07:35.109355 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:35.109382 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:35.109497 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:35.110570 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:35.110575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:35.111042 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:35.111209 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.111219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.112336 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.112345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.113354 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.113367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.114304 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.114310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.114555 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.115161 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:35.115232 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:35.118530 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":317,"threads_started":1}
W20260925 11:07:35.119179 18158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:35.108863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:53367 (local address 127.8.181.62:43081)
0925 11:07:35.110186 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:35.110192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:35.110209 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:35.110261 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:35.110266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:35.110373 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:35.110547 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:35.111364 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.112190 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.113509 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.114138 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.114275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.118854 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:53367: BlockingRecv error: recv got EOF from 127.8.181.3:53367 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1209,"thread_start_us":1020,"threads_started":1}
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:35 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:35.615000 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:35.603918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:42403)
0925 11:07:35.604688 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:35.604726 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:35.604931 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:35.605445 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:35.605450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:35.605982 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:35.606324 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.606340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.608246 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.608252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.609396 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:35.609413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.610307 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.610312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:35.610452 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.611188 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:35.611252 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:35.614721 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":400,"threads_started":1}
W20260925 11:07:35.615268 18158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:35.604316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42403 (local address 127.8.181.62:43081)
0925 11:07:35.604534 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:35.604542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:35.604956 (+ 414us) server_negotiation.cc:407] Connection header received
0925 11:07:35.605114 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:35.605120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:35.605215 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:35.605413 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:35.606786 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.608021 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.609634 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:35.610099 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:35.610234 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:35.615025 (+ 4791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:42403: BlockingRecv error: recv got EOF from 127.8.181.2:42403 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:36.137017 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:36.120027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:55583)
0925 11:07:36.124359 (+ 4332us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:36.124386 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:36.124567 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:36.125334 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:36.125341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:36.125877 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:36.126077 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.126092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.127769 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.127775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.129749 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.129771 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.131209 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.131216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.131441 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.132363 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:36.132392 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:36.136641 (+ 4249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3448,"thread_start_us":3302,"threads_started":1}
I20260925 11:07:36.137264 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:36.119573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:51675)
0925 11:07:36.125029 (+ 5456us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:36.125049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:36.125183 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:36.125608 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:36.125616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:36.126141 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:36.126380 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.126392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.127900 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.127909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.129648 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.129668 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.131209 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.131216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.131456 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.132371 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:36.132396 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:36.136869 (+ 4473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5314,"thread_start_us":5155,"threads_started":1}
W20260925 11:07:36.137328 18163 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:36.120184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55583 (local address 127.8.181.62:43081)
0925 11:07:36.124586 (+ 4402us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:36.124880 (+ 294us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:36.124897 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:36.124955 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:36.124966 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:36.125060 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:36.125208 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:36.126253 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.127556 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.129923 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.130990 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.131185 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.136985 (+ 5800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:55583: BlockingRecv error: recv got EOF from 127.8.181.3:55583 (error 108)
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":4258,"spinlock_wait_cycles":9984,"thread_start_us":4106,"threads_started":1}
W20260925 11:07:36.138384 18160 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:36.119659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:51675 (local address 127.8.181.62:43081)
0925 11:07:36.124107 (+ 4448us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:36.124117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:36.125113 (+ 996us) server_negotiation.cc:407] Connection header received
0925 11:07:36.125335 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:36.125340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:36.125432 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:36.125549 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:36.126555 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.127750 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.129923 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.130990 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.131182 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.137960 (+ 6778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:51675: BlockingRecv error: recv got EOF from 127.8.181.1:51675 (error 108)
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":4263,"spinlock_wait_cycles":896,"thread_start_us":4380,"threads_started":1}
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:36 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:36.629011 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:36.616525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:39759)
0925 11:07:36.617414 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:36.617438 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:36.617581 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:36.618150 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:36.618156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:36.618907 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:36.619153 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.619166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.620667 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.620672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.621787 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:36.621805 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.622918 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.622924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:36.623124 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.624050 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:36.624078 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:36.628746 (+ 4668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":532,"threads_started":1}
W20260925 11:07:36.629185 18160 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:36.616901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:39759 (local address 127.8.181.62:43081)
0925 11:07:36.617203 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:36.617208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:36.617566 (+ 358us) server_negotiation.cc:407] Connection header received
0925 11:07:36.617812 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:36.617817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:36.617887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:36.617989 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:36.619524 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.620382 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.622086 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:36.622728 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:36.622890 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:36.628974 (+ 6084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:39759: BlockingRecv error: recv got EOF from 127.8.181.2:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:37.150173 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:37.138527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:44479)
0925 11:07:37.139092 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:37.139128 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:37.139292 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:37.140205 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:37.140210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:37.140847 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:37.141020 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.141031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.142546 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.142552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.143689 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.143704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.144829 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.144835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.145034 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.145921 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:37.145979 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:37.149799 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":271,"threads_started":1}
I20260925 11:07:37.150345 18166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:37.138577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:37465)
0925 11:07:37.139363 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:37.139380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:37.139479 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:37.140690 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:37.140695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:37.141057 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:37.141312 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.141324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.143092 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.143097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.144324 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.144343 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.145909 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.145916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.146130 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.146799 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:37.146822 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:37.150027 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":550,"threads_started":1}
W20260925 11:07:37.150519 18167 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:37.138607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:44479 (local address 127.8.181.62:43081)
0925 11:07:37.139731 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:37.139738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:37.139761 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:07:37.139842 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:37.139849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:37.139963 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:37.140090 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:37.141195 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.142280 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.143862 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.144547 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.144745 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.150060 (+ 5315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:44479: BlockingRecv error: recv got EOF from 127.8.181.3:44479 (error 108)
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":1001,"thread_start_us":889,"threads_started":1}
W20260925 11:07:37.150524 18168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:37.139650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:37465 (local address 127.8.181.62:43081)
0925 11:07:37.140318 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:37.140326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:37.140345 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:07:37.140388 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:37.140394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:37.140491 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:37.140625 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:37.141509 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.142908 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.144527 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.145522 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.145721 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.150262 (+ 4541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:37465: BlockingRecv error: recv got EOF from 127.8.181.1:37465 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":552,"thread_start_us":280,"threads_started":1}
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:37 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:37.639452 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:37.630215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:43659)
0925 11:07:37.630929 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:37.630951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:37.631131 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:37.631534 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:37.631538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:37.631955 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:37.632176 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.632186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.633248 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.633252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.634347 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:37.634361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.635302 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.635307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:37.635440 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.636198 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:37.636218 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:37.639214 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":467,"threads_started":1}
W20260925 11:07:37.639801 18168 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:37.630429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:43659 (local address 127.8.181.62:43081)
0925 11:07:37.630744 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:37.630749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:37.631071 (+ 322us) server_negotiation.cc:407] Connection header received
0925 11:07:37.631285 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:37.631288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:37.631353 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:37.631439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:37.632334 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.633103 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.634606 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:37.635116 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:37.635226 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:37.639485 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:43659: BlockingRecv error: recv got EOF from 127.8.181.2:43659 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:38.188063 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:38.151642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:47335)
0925 11:07:38.166992 (+ 15350us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:38.167019 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:38.167149 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:38.167798 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:38.167803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:38.168332 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:38.168562 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.168575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.170257 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.170263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.172121 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.172143 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.173376 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.173385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.173603 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.183848 (+ 10245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:38.183873 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:38.187815 (+ 3942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15239,"thread_start_us":15123,"threads_started":1}
I20260925 11:07:38.188748 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:38.151581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:41937)
0925 11:07:38.166812 (+ 15231us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:38.166838 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:38.167004 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:38.168220 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:38.168225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:38.168807 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:38.168977 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.168987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.170439 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.170448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.172262 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.172282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.173541 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.173546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.173674 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.183903 (+ 10229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:38.183925 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:38.188361 (+ 4436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15078,"thread_start_us":15450,"threads_started":1}
W20260925 11:07:38.188783 18171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:38.151665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:47335 (local address 127.8.181.62:43081)
0925 11:07:38.166600 (+ 14935us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:38.166607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:38.167278 (+ 671us) server_negotiation.cc:407] Connection header received
0925 11:07:38.167348 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:38.167355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:38.167475 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:38.167601 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:38.168806 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.170033 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.172374 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.173180 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.173300 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.188323 (+ 15023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:47335: BlockingRecv error: recv got EOF from 127.8.181.3:47335 (error 108)
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":14759,"thread_start_us":14635,"threads_started":1}
W20260925 11:07:38.189353 18173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:38.151786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:41937 (local address 127.8.181.62:43081)
0925 11:07:38.167662 (+ 15876us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:38.167667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:38.167681 (+ 14us) server_negotiation.cc:407] Connection header received
0925 11:07:38.167736 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:38.167740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:38.167803 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:38.167903 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:38.169151 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.170270 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.172416 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.173355 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.173499 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.188881 (+ 15382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:41937: BlockingRecv error: recv got EOF from 127.8.181.1:41937 (error 108)
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":15776,"thread_start_us":15713,"threads_started":1}
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:38 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:38.652236 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:38.640810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:40231)
0925 11:07:38.641956 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:38.641986 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:38.642154 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:38.642580 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:38.642586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:38.643225 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:38.643396 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.643405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.644705 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.644711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.645894 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:38.645907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.647009 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.647017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:38.647209 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.648209 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:38.648235 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:38.651905 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":460,"threads_started":1}
W20260925 11:07:38.652860 18173 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:38.640908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:40231 (local address 127.8.181.62:43081)
0925 11:07:38.641203 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:38.641210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:38.642159 (+ 949us) server_negotiation.cc:407] Connection header received
0925 11:07:38.642269 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:38.642275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:38.642365 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:38.642662 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:38.643605 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.644531 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.646084 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:38.646650 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:38.646911 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:38.652584 (+ 5673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:40231: BlockingRecv error: recv got EOF from 127.8.181.2:40231 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:39.200937 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:39.189340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:52453)
0925 11:07:39.191008 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:39.191045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:39.191215 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:39.191739 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:39.191747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:39.192373 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:39.192615 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.192632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.194165 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.194170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.195339 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.195355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.196135 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.196141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.196317 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.197014 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:39.197042 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:39.200512 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1451,"thread_start_us":1183,"threads_started":1}
I20260925 11:07:39.201017 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:39.189903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:58849)
0925 11:07:39.192189 (+ 2286us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:39.192213 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:39.192336 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:39.192703 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:39.192710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:39.193233 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:39.193400 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.193410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.194827 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.194831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.195751 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.195763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.196603 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.196609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.196785 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.197330 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:39.197349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:39.200673 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2125,"thread_start_us":2010,"threads_started":1}
W20260925 11:07:39.201174 18178 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:39.191163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:58849 (local address 127.8.181.62:43081)
0925 11:07:39.192425 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:39.192432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:39.192449 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:07:39.192500 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:39.192503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:39.192549 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:39.192664 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:39.193606 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.194661 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.195912 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.196408 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.196545 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.200903 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:58849: BlockingRecv error: recv got EOF from 127.8.181.1:58849 (error 108)
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":381,"threads_started":1}
W20260925 11:07:39.201251 18176 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:39.189490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52453 (local address 127.8.181.62:43081)
0925 11:07:39.191199 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:39.191205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:39.191229 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:07:39.191394 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:39.191403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:39.191552 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:39.191697 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:39.192802 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.193970 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.195524 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.195957 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.196121 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.200779 (+ 4658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:52453: BlockingRecv error: recv got EOF from 127.8.181.3:52453 (error 108)
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":1499,"threads_started":1}
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:39 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:39.664565 18179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:39.653415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:58101)
0925 11:07:39.653926 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:39.653947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:39.654070 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:39.654399 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:39.654403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:39.654907 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:39.655070 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.655078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.656803 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.656808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.657973 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:39.657989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.659093 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.659099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:39.659288 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.660330 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:39.660367 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:39.664280 (+ 3913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
W20260925 11:07:39.664952 18176 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:39.653643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:58101 (local address 127.8.181.62:43081)
0925 11:07:39.653863 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:39.653896 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:39.654055 (+ 159us) server_negotiation.cc:407] Connection header received
0925 11:07:39.654192 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:39.654196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:39.654264 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:39.654355 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:39.655272 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.656560 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.658199 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:39.658865 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:39.659038 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:39.664596 (+ 5558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:58101: BlockingRecv error: recv got EOF from 127.8.181.2:58101 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:40.212159 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:40.202438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:39479)
0925 11:07:40.203228 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:40.203252 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:40.203369 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:40.203688 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:40.203691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:40.204020 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:40.204169 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.204177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.205379 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.205383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.206284 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.206295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.207269 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.207274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.207432 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.208194 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:40.208211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:40.211865 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":522,"threads_started":1}
I20260925 11:07:40.212180 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:40.202032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:46047)
0925 11:07:40.202718 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:40.202744 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:40.202879 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:40.203447 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:40.203453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:40.203961 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:40.204135 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.204144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.205240 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.205246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.206234 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.206246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.207187 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.207194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.207401 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.208009 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:40.208028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:40.211810 (+ 3782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":366,"threads_started":1}
W20260925 11:07:40.212383 18183 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:40.202853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:39479 (local address 127.8.181.62:43081)
0925 11:07:40.203377 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:40.203383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:40.203401 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:07:40.203461 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:40.203465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:40.203530 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:40.203641 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:40.204359 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.205240 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.206514 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.207055 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.207248 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.212076 (+ 4828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:39479: BlockingRecv error: recv got EOF from 127.8.181.3:39479 (error 108)
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":384,"thread_start_us":314,"threads_started":1}
W20260925 11:07:40.212630 18181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:40.202266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:46047 (local address 127.8.181.62:43081)
0925 11:07:40.203039 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:40.203046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:40.203067 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:40.203162 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:40.203169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:40.203268 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:40.203390 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:40.204293 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.205092 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.206389 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.207024 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.207163 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.212069 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:46047: BlockingRecv error: recv got EOF from 127.8.181.1:46047 (error 108)
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":635,"spinlock_wait_cycles":16768,"thread_start_us":520,"threads_started":1}
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:40 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:40.676417 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:40.665821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:47015)
0925 11:07:40.666418 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:40.666441 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:40.666592 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:40.666963 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:40.666967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:40.667403 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:40.667570 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.667581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.668807 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.668812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.670226 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:40.670240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.671270 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.671275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:40.671417 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.672149 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:40.672175 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:40.675900 (+ 3725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":323,"threads_started":1}
W20260925 11:07:40.676766 18181 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:40.665970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:47015 (local address 127.8.181.62:43081)
0925 11:07:40.666186 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:40.666190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:40.666593 (+ 403us) server_negotiation.cc:407] Connection header received
0925 11:07:40.666686 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:40.666689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:40.666758 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:40.666876 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:40.667765 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.668644 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.670443 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:40.671054 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:40.671176 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:40.676300 (+ 5124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:47015: BlockingRecv error: recv got EOF from 127.8.181.2:47015 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:41.237957 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:41.213446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:38083)
0925 11:07:41.227809 (+ 14363us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:41.227837 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:41.228006 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:41.228779 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:41.228783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:41.229161 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:41.229309 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.229318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.230578 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.230584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.231638 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.231654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.232638 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.232644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.232835 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.233656 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:41.233692 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:41.237590 (+ 3898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14193,"thread_start_us":14040,"threads_started":1}
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:41.238931 18186 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:41.213567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:38083 (local address 127.8.181.62:43081)
0925 11:07:41.228220 (+ 14653us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:41.228231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:41.228257 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:07:41.228328 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:41.228333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:41.228443 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:41.228576 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:41.229499 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.230372 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.231796 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.232468 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.232608 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.238028 (+ 5420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:38083: BlockingRecv error: recv got EOF from 127.8.181.1:38083 (error 108)
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":14489,"spinlock_wait_cycles":8448,"thread_start_us":14312,"threads_started":1}
I20260925 11:07:41.239763 18188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:41.213652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:46731)
0925 11:07:41.227310 (+ 13658us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:41.227347 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:41.227537 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:41.228916 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:41.228920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:41.229292 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:41.229515 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.229541 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.230977 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.230981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.232102 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.232120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.233488 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.233495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.234318 (+ 823us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.235018 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:41.235046 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:41.239426 (+ 4380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13470,"thread_start_us":13402,"threads_started":1}
W20260925 11:07:41.239961 18187 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:41.213603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:46731 (local address 127.8.181.62:43081)
0925 11:07:41.228361 (+ 14758us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:41.228366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:41.228380 (+ 14us) server_negotiation.cc:407] Connection header received
0925 11:07:41.228414 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:41.228418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:41.228460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:41.228576 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:41.229769 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.230733 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.232325 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.233060 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.233453 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.239650 (+ 6197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:46731: BlockingRecv error: recv got EOF from 127.8.181.3:46731 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":14647,"spinlock_wait_cycles":7808,"thread_start_us":14506,"threads_started":1}
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:41 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:41.691110 18189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:41.678095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:43951)
0925 11:07:41.678731 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:41.678757 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:41.678885 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:41.679288 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:41.679293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:41.679766 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:41.679925 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.679934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.681521 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.681529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.682948 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:41.682971 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.684020 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.684027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:41.684186 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.685116 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:41.685139 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:41.690712 (+ 5573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":398,"threads_started":1}
W20260925 11:07:41.691282 18187 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:41.678654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:43951 (local address 127.8.181.62:43081)
0925 11:07:41.678857 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:41.678863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:41.678884 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:41.679009 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:41.679013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:41.679090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:41.679190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:41.680152 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.681176 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.683238 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:41.683719 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:41.683881 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:41.691018 (+ 7137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:43951: BlockingRecv error: recv got EOF from 127.8.181.2:43951 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:42.254045 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:42.239458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:42381)
0925 11:07:42.240207 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:42.240231 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:42.240405 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:42.241218 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:42.241226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:42.242414 (+ 1188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:42.242818 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.242830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.244466 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.244476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.246665 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.246688 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.247963 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.247971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.248188 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.249173 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:42.249203 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:42.253630 (+ 4427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":477,"threads_started":1}
I20260925 11:07:42.254096 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:42.240697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:46723)
0925 11:07:42.241211 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:42.241238 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:42.241366 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:42.242221 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:42.242227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:42.242766 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:42.242920 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.242929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.244294 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.244300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.245987 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.246013 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.247681 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.247689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.247896 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.248790 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:42.248831 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:42.253384 (+ 4553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":329,"threads_started":1}
W20260925 11:07:42.254153 18193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:42.240857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:46723 (local address 127.8.181.62:43081)
0925 11:07:42.241930 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:42.241934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:42.241948 (+ 14us) server_negotiation.cc:407] Connection header received
0925 11:07:42.241986 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:42.241994 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:42.242062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:42.242187 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:42.243170 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.244128 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.246365 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.247342 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.247640 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.253838 (+ 6198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:46723: BlockingRecv error: recv got EOF from 127.8.181.3:46723 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":943,"thread_start_us":829,"threads_started":1}
W20260925 11:07:42.254997 18191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:42.239630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:42381 (local address 127.8.181.62:43081)
0925 11:07:42.240610 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:42.240618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:42.240640 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:07:42.240708 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:42.240714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:42.240796 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:42.240922 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:42.243057 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.244038 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.246840 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.247708 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.247933 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.254634 (+ 6701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:42381: BlockingRecv error: recv got EOF from 127.8.181.1:42381 (error 108)
Metrics: {"mutex_wait_us":214,"server-negotiator.queue_time_us":836,"thread_start_us":1052,"threads_started":1}
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:42 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:42.706574 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:42.693060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:42437)
0925 11:07:42.693869 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:42.693893 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:42.694062 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:42.694954 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:42.694961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:42.695542 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:42.695748 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.695758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.697417 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.697427 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.699288 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:42.699316 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.700701 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.700710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:42.700944 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.702126 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:42.702148 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:42.706141 (+ 3993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":399,"threads_started":1}
W20260925 11:07:42.707135 18191 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:42.693172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42437 (local address 127.8.181.62:43081)
0925 11:07:42.693475 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:42.693486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:42.694040 (+ 554us) server_negotiation.cc:407] Connection header received
0925 11:07:42.694286 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:42.694291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:42.694363 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:42.694509 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:42.695968 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.697058 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.699517 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:42.700272 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:42.700526 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:42.706816 (+ 6290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:42437: BlockingRecv error: recv got EOF from 127.8.181.2:42437 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:43.279845 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:43.256121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:41081)
0925 11:07:43.257066 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:43.257093 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:43.257290 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:43.258691 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:43.258697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:43.259318 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:43.259568 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.259584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.261893 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.261900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.263877 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.263904 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.265865 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.265874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.266073 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.274362 (+ 8289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:43.274395 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:43.279378 (+ 4983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":850,"threads_started":1}
I20260925 11:07:43.279963 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:43.255410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:51163)
0925 11:07:43.257261 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:43.257354 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:43.257620 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:43.258872 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:43.258878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:43.259457 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:43.259755 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.259775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.262295 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.262302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.263903 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.263929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.265813 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.265822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.266027 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.274362 (+ 8335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:43.274395 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:43.279590 (+ 5195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":667,"threads_started":1}
W20260925 11:07:43.280104 18198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:43.256825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:41081 (local address 127.8.181.62:43081)
0925 11:07:43.258141 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:43.258146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:43.258161 (+ 15us) server_negotiation.cc:407] Connection header received
0925 11:07:43.258251 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:43.258257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:43.258360 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:43.258586 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:43.259781 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.261605 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.264241 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.265491 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.265740 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.279726 (+ 13986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:41081: BlockingRecv error: recv got EOF from 127.8.181.1:41081 (error 108)
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":1157,"thread_start_us":908,"threads_started":1}
W20260925 11:07:43.280157 18196 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:43.255991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:51163 (local address 127.8.181.62:43081)
0925 11:07:43.256813 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:43.256825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:43.257534 (+ 709us) server_negotiation.cc:407] Connection header received
0925 11:07:43.258465 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:43.258505 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:43.258596 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:43.258693 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:43.259935 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.261605 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.264241 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.265491 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.265680 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.279823 (+ 14143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:51163: BlockingRecv error: recv got EOF from 127.8.181.3:51163 (error 108)
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":648,"thread_start_us":482,"threads_started":1}
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:43 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:43.717759 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:43.707861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:46317)
0925 11:07:43.708755 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:43.708946 (+ 191us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:43.709178 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:43.709633 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:43.709637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:43.710097 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:43.710255 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.710263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.711462 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.711470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.712460 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:43.712472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.713245 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.713251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:43.713458 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.714193 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:43.714212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:43.717395 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":373,"threads_started":1}
W20260925 11:07:43.717931 18196 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:43.707961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:46317 (local address 127.8.181.62:43081)
0925 11:07:43.708566 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:43.708576 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:43.709146 (+ 570us) server_negotiation.cc:407] Connection header received
0925 11:07:43.709380 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:43.709384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:43.709468 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:43.709589 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:43.710427 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.711301 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.712648 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:43.713077 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:43.713223 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:43.717652 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:46317: BlockingRecv error: recv got EOF from 127.8.181.2:46317 (error 108)
Metrics: {"server-negotiator.queue_time_us":238}
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:44.302058 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:44.281686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:46343)
0925 11:07:44.287029 (+ 5343us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:44.287063 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:44.287339 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:44.288171 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:44.288177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:44.288842 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:44.289297 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.289318 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.291342 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.291349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.293030 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.293053 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.295067 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.295074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.295268 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.296264 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:44.296299 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:44.301651 (+ 5352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4893,"thread_start_us":4629,"threads_started":1}
I20260925 11:07:44.302398 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:44.282686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:32873)
0925 11:07:44.287309 (+ 4623us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:44.287335 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:44.287456 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:44.288781 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:44.288788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:44.289219 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:44.289753 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.289858 (+ 105us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.291493 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.291499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.292774 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.292793 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.295020 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.295031 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.295268 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.296264 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:44.296299 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:44.302025 (+ 5726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4479,"thread_start_us":4192,"threads_started":1}
W20260925 11:07:44.302956 18201 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:44.281914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:46343 (local address 127.8.181.62:43081)
0925 11:07:44.286293 (+ 4379us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:44.286305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:44.287483 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 11:07:44.287569 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:44.287575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:44.287892 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:44.288153 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:44.289860 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.291168 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.293311 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.294744 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.295003 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.302236 (+ 7233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:46343: BlockingRecv error: recv got EOF from 127.8.181.1:46343 (error 108)
Metrics: {"mutex_wait_us":322,"server-negotiator.queue_time_us":3964,"spinlock_wait_cycles":45824,"thread_start_us":5458,"threads_started":1}
W20260925 11:07:44.303088 18202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:44.282215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:32873 (local address 127.8.181.62:43081)
0925 11:07:44.288075 (+ 5860us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:44.288080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:44.288096 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:07:44.288141 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:44.288144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:44.288223 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:44.288668 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:44.290084 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.291328 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.293116 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.294720 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.295004 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.302532 (+ 7528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:32873: BlockingRecv error: recv got EOF from 127.8.181.3:32873 (error 108)
Metrics: {"mutex_wait_us":269,"server-negotiator.queue_time_us":5660,"spinlock_wait_cycles":11648,"thread_start_us":3550,"threads_started":1}
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:44 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:44.732158 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:44.719324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:40351)
0925 11:07:44.720081 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:44.720104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:44.720236 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:44.720764 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:44.720768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:44.721211 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:44.721383 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.721394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.722939 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.722949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.724019 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:44.724034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.725653 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.725662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:44.726123 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.727614 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:44.727638 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:44.731875 (+ 4237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":460,"threads_started":1}
W20260925 11:07:44.732694 18202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:44.719745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:40351 (local address 127.8.181.62:43081)
0925 11:07:44.720033 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:44.720040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:44.720239 (+ 199us) server_negotiation.cc:407] Connection header received
0925 11:07:44.720336 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:44.720340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:44.720410 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:44.720520 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:44.721607 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.722609 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.724355 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:44.725263 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:44.725462 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:44.732405 (+ 6943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:40351: BlockingRecv error: recv got EOF from 127.8.181.2:40351 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:45.316946 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:45.303885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:50641)
0925 11:07:45.304520 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:45.304542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:45.304723 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:45.305233 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:45.305238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:45.305792 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:45.306051 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.306066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.307603 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.307610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.309000 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.309019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.310081 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.310089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.310308 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.311658 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:45.311689 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:45.316505 (+ 4816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":352,"threads_started":1}
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 11:07:45.317690 18206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:45.304035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:50641 (local address 127.8.181.62:43081)
0925 11:07:45.304911 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:45.304916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:45.304932 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:07:45.304986 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:45.304989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:45.305069 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:45.305199 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:45.306293 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.307409 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.309320 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.309875 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.310019 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.317373 (+ 7354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:50641: BlockingRecv error: recv got EOF from 127.8.181.1:50641 (error 108)
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":576,"threads_started":1}
I20260925 11:07:45.319061 18207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:45.304083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:46575)
0925 11:07:45.305843 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:45.305875 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:45.306008 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:45.306832 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:45.306837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:45.307279 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:45.307455 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.307466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.309270 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.309276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.310571 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.310597 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.312176 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.312187 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.312454 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.313432 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:45.313468 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:45.318689 (+ 5221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1621,"thread_start_us":1451,"threads_started":1}
W20260925 11:07:45.319617 18208 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:45.304865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:46575 (local address 127.8.181.62:43081)
0925 11:07:45.306340 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:45.306347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:45.306366 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:07:45.306451 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:45.306458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:45.306595 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:45.306746 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:45.307733 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.308811 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.310922 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.311737 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.311862 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.319324 (+ 7462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:46575: BlockingRecv error: recv got EOF from 127.8.181.3:46575 (error 108)
Metrics: {"server-negotiator.queue_time_us":1330,"thread_start_us":1029,"threads_started":1}
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:45 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:45.749909 18209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:45.733842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:51603)
0925 11:07:45.735464 (+ 1622us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:45.735546 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:45.735727 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:45.736324 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:45.736328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:45.736751 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:45.736953 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.736963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.739217 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.739237 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.740877 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:45.740899 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.742331 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.742339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:45.742622 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.744060 (+ 1438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:45.744087 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:45.748921 (+ 4834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1412,"thread_start_us":967,"threads_started":1}
W20260925 11:07:45.750866 18208 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:45.734026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:51603 (local address 127.8.181.62:43081)
0925 11:07:45.734250 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:45.734256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:45.735696 (+ 1440us) server_negotiation.cc:407] Connection header received
0925 11:07:45.735944 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:45.735951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:45.736050 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:45.736172 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:45.737187 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.738007 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.741346 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:45.741838 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:45.742099 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:45.749933 (+ 7834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:51603: BlockingRecv error: recv got EOF from 127.8.181.2:51603 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.3@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.1@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:46.334618 18212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:46.320730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.3:43065)
0925 11:07:46.321353 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:46.321384 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:46.321562 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:46.322016 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:46.322021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:46.322586 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:46.322816 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.322828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.324545 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.324552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.325985 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.326007 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.327186 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.327199 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.327455 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.328218 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:46.328244 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:46.334121 (+ 5877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
W20260925 11:07:46.335104 18213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:46.320980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:43065 (local address 127.8.181.62:43081)
0925 11:07:46.321623 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:46.321629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:46.321652 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:07:46.321740 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:46.321746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:46.321848 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:46.321991 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:46.323233 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.324315 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.326177 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.326956 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.327078 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.334570 (+ 7492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.3:43065: BlockingRecv error: recv got EOF from 127.8.181.3:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":355,"threads_started":1}
I20260925 11:07:46.336145 18210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:46.318620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.1:58591)
0925 11:07:46.319402 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:46.319420 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:46.319604 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:46.320354 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:46.320369 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:46.321108 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:46.321419 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.321438 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.323587 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.323596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.325495 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.325527 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.326874 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.326880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.327131 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.328197 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:46.328224 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:46.335733 (+ 7509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":514,"threads_started":1}
W20260925 11:07:46.336383 18211 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:46.319336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:58591 (local address 127.8.181.62:43081)
0925 11:07:46.319901 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:46.319907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:46.319926 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:07:46.319996 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:46.320000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:46.320067 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:46.320200 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:46.321731 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.323294 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.325861 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.326626 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.326807 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.336059 (+ 9252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.1:58591: BlockingRecv error: recv got EOF from 127.8.181.1:58591 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":326,"threads_started":1}
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 11:07:46 dist-test-slave-6042 krb5kdc[17138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.181.2@KRBTEST.COM for oryx/127.8.181.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 11:07:46.766122 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:46.751524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:43081 (local address 127.8.181.2:34317)
0925 11:07:46.752210 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:46.752231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:46.752364 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:46.753025 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:46.753031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:46.753509 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:46.753775 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.753786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.755764 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.755774 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.757454 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:46.757487 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.758944 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.758958 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:46.759225 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.760336 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:46.760359 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:46.765697 (+ 5338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.181.62:43081: Server oryx/127.8.181.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":602,"threads_started":1}
W20260925 11:07:46.766409 18211 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:46.751832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:34317 (local address 127.8.181.62:43081)
0925 11:07:46.752088 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:46.752094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:46.752346 (+ 252us) server_negotiation.cc:407] Connection header received
0925 11:07:46.752567 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:46.752571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:46.752641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:46.752744 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:46.754143 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.755355 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.757801 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:46.758698 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:46.758887 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:46.766092 (+ 7205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.181.2:34317: BlockingRecv error: recv got EOF from 127.8.181.2:34317 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 11:07:47.042538 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17727
I20260925 11:07:47.060189 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:47.044288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.8.181.1:33971)
0925 11:07:47.044884 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:47.044892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:47.044923 (+ 31us) server_negotiation.cc:407] Connection header received
0925 11:07:47.045232 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:47.045244 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:47.045724 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:47.045967 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:47.048608 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.050037 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.051678 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.052387 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.052514 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:47.055574 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:47.055584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:47.055600 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:47.055664 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:47.059609 (+ 3945us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 11:07:47.059939 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46014: Request ticket server kudu/127.8.181.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
W20260925 11:07:47.060325 18216 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46014: Request ticket server kudu/127.8.181.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 11:07:47.061208 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17727: Not authorized: Client connection negotiation failed: client connection to 127.8.181.1:33971: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 11:07:47.061503 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17727
I20260925 11:07:47.083765 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17878
I20260925 11:07:47.103719 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:47.085792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.8.181.2:44303)
0925 11:07:47.086821 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:47.086831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:47.087176 (+ 345us) server_negotiation.cc:407] Connection header received
0925 11:07:47.087285 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:47.087293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:47.087599 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:47.087810 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:47.090286 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.091697 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.093287 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.094459 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.094881 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:47.098708 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:47.098720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:47.098737 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:47.098802 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:47.102354 (+ 3552us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 11:07:47.103280 (+ 926us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46398: Request ticket server kudu/127.8.181.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":443,"threads_started":1}
W20260925 11:07:47.104024 18217 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46398: Request ticket server kudu/127.8.181.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 11:07:47.104175 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17878: Not authorized: Client connection negotiation failed: client connection to 127.8.181.2:44303: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 11:07:47.104364 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17878
I20260925 11:07:47.122792 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18009
I20260925 11:07:47.140151 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:47.124235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.8.181.3:43205)
0925 11:07:47.125010 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:47.125018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:47.125050 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:07:47.125115 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:47.125122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:47.125363 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:47.125538 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:47.127730 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.128976 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.130705 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.131448 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.131588 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:47.135631 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:47.135698 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:47.135715 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:47.135780 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:47.139522 (+ 3742us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 11:07:47.139880 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51212: Request ticket server kudu/127.8.181.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
W20260925 11:07:47.140316 18218 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51212: Request ticket server kudu/127.8.181.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 11:07:47.140502 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18009: Not authorized: Client connection negotiation failed: client connection to 127.8.181.3:43205: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.181.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 11:07:47.140751 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18009
I20260925 11:07:47.158880 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 17658
I20260925 11:07:47.161056 17720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:47.297538 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 17658
2026-09-25T11:07:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21689 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T11:07:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:47Z Disabled control of system clock
I20260925 11:07:47.399350 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40365
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 11:07:47.828058 18223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:47.828423 18223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:47.828573 18223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:47.836694 18223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:47.836846 18223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:47.836895 18223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:47.836982 18223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:47.837047 18223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:47.848311 18223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:47.850000 18223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:47.852912 18223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:47.873543 18231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:47.873518 18228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:47.873646 18223 server_base.cc:1034] running on GCE node
W20260925 11:07:47.873821 18229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:47.875329 18223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:47.877259 18223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:47.880854 18223 hybrid_clock.cc:648] HybridClock initialized: now 1790334467880568 us; error 246 us; skew 500 ppm
I20260925 11:07:47.881748 18223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:47.884940 18223 webserver.cc:460] Webserver started at http://127.8.181.62:42025/ using document root <none> and password file <none>
I20260925 11:07:47.885847 18223 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:47.886013 18223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:47.886384 18223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:47.889842 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "b6b528c1be7041af868e50ddc6beafc3"
format_stamp: "Formatted at 2026-09-25 11:07:47 on dist-test-slave-6042"
server_key: "f29bc9c394a762d38767a98b2120b8cb"
server_key_iv: "6c64514d8f968b9fafa14150845ed5a1"
server_key_version: "encryptionkey@0"
I20260925 11:07:47.890972 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "b6b528c1be7041af868e50ddc6beafc3"
format_stamp: "Formatted at 2026-09-25 11:07:47 on dist-test-slave-6042"
server_key: "f29bc9c394a762d38767a98b2120b8cb"
server_key_iv: "6c64514d8f968b9fafa14150845ed5a1"
server_key_version: "encryptionkey@0"
I20260925 11:07:47.897435 18223 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 11:07:47.903352 18237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:47.905886 18223 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260925 11:07:47.906211 18223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "b6b528c1be7041af868e50ddc6beafc3"
format_stamp: "Formatted at 2026-09-25 11:07:47 on dist-test-slave-6042"
server_key: "f29bc9c394a762d38767a98b2120b8cb"
server_key_iv: "6c64514d8f968b9fafa14150845ed5a1"
server_key_version: "encryptionkey@0"
I20260925 11:07:47.906605 18223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:47.936729 18223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:47.937896 18223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:47.938288 18223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:47.968730 18223 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:40365
I20260925 11:07:47.968621 18288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:40365 every 8 connection(s)
I20260925 11:07:47.970367 18223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:47.973178 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18223
I20260925 11:07:47.973366 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:47.974135 8916 external_mini_cluster.cc:1428] Setting key d8b1e3e9be8d48f9ad4d83a10b0a92e1
I20260925 11:07:47.978042 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:47.998348 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:47.976988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.8.181.62:40365)
0925 11:07:47.978365 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:47.978384 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:47.978452 (+ 68us) server_negotiation.cc:407] Connection header received
0925 11:07:47.981208 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:47.981406 (+ 198us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:47.982316 (+ 910us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:47.983006 (+ 690us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:47.984199 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.986767 (+ 2568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.989674 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:47.991494 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:47.991780 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:47.994079 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:47.994100 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:47.994136 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:47.995499 (+ 1363us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:47.995651 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:47.995699 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:47.995889 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:47.996289 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 11:07:47.996774 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":448,"threads_started":1}
I20260925 11:07:47.998644 18289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3: Bootstrap starting.
I20260925 11:07:48.003350 18289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:48.004601 18289 log.cc:824] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:48.009938 18289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3: No bootstrap required, opened a new log
I20260925 11:07:48.019028 18289 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } }
I20260925 11:07:48.019753 18289 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:48.019899 18289 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b528c1be7041af868e50ddc6beafc3, State: Initialized, Role: FOLLOWER
I20260925 11:07:48.020711 18289 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } }
I20260925 11:07:48.021040 18289 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:48.021229 18289 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:48.021430 18289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:48.024104 18289 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } }
I20260925 11:07:48.024780 18289 leader_election.cc:302] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b528c1be7041af868e50ddc6beafc3; no voters:
I20260925 11:07:48.025578 18289 leader_election.cc:288] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:48.025875 18294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:48.026809 18294 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [term 1 LEADER]: Becoming Leader. State: Replica: b6b528c1be7041af868e50ddc6beafc3, State: Running, Role: LEADER
I20260925 11:07:48.027707 18294 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } }
I20260925 11:07:48.028363 18289 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:48.031440 18295 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b528c1be7041af868e50ddc6beafc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } } }
I20260925 11:07:48.031487 18296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b528c1be7041af868e50ddc6beafc3. Latest consensus state: current_term: 1 leader_uuid: "b6b528c1be7041af868e50ddc6beafc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b528c1be7041af868e50ddc6beafc3" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40365 } } }
I20260925 11:07:48.032435 18295 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:48.032444 18296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:48.040915 18302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:48.047586 18302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:48.058645 18302 catalog_manager.cc:1347] Generated new cluster ID: 04ffe60838c44dd19f45024413a234b9
I20260925 11:07:48.058876 18302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:48.088819 18302 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:48.091365 18302 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:48.107749 18302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6b528c1be7041af868e50ddc6beafc3: Generated new TSK 0
I20260925 11:07:48.109211 18302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:07:48.129513 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--builtin_ntp_servers=127.8.181.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 11:07:48.639664 18313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:48.640074 18313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:48.640292 18313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:48.650352 18313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:48.650774 18313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:48.651042 18313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:48.664398 18313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:48.666347 18313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:48.669732 18313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:48.691298 18318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:48.692193 18319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:48.692466 18321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:48.693988 18313 server_base.cc:1034] running on GCE node
I20260925 11:07:48.694955 18313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:48.696918 18313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:48.698585 18313 hybrid_clock.cc:648] HybridClock initialized: now 1790334468698324 us; error 159 us; skew 500 ppm
I20260925 11:07:48.699280 18313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:48.710350 18313 webserver.cc:460] Webserver started at http://127.8.181.1:46469/ using document root <none> and password file <none>
I20260925 11:07:48.711390 18313 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:48.711572 18313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:48.711963 18313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:48.716594 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "375287d0f98f47669c17d4fb86e0648d"
format_stamp: "Formatted at 2026-09-25 11:07:48 on dist-test-slave-6042"
server_key: "d8879c067ce0f2313b07128d19c5af74"
server_key_iv: "ef1c86bacbb1793dc405eacf44f0424a"
server_key_version: "encryptionkey@0"
I20260925 11:07:48.717904 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "375287d0f98f47669c17d4fb86e0648d"
format_stamp: "Formatted at 2026-09-25 11:07:48 on dist-test-slave-6042"
server_key: "d8879c067ce0f2313b07128d19c5af74"
server_key_iv: "ef1c86bacbb1793dc405eacf44f0424a"
server_key_version: "encryptionkey@0"
I20260925 11:07:48.725325 18313 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 11:07:48.731086 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:48.733563 18313 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260925 11:07:48.733880 18313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "375287d0f98f47669c17d4fb86e0648d"
format_stamp: "Formatted at 2026-09-25 11:07:48 on dist-test-slave-6042"
server_key: "d8879c067ce0f2313b07128d19c5af74"
server_key_iv: "ef1c86bacbb1793dc405eacf44f0424a"
server_key_version: "encryptionkey@0"
I20260925 11:07:48.734198 18313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:48.760695 18313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:48.762537 18313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:48.763110 18313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:48.764850 18313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:48.767565 18313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:48.767760 18313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:48.767930 18313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:48.768030 18313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:48.825784 18313 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:39783
I20260925 11:07:48.825853 18439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:39783 every 8 connection(s)
I20260925 11:07:48.828665 18313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:48.831084 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18313
I20260925 11:07:48.831384 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:48.833945 8916 external_mini_cluster.cc:1428] Setting key f2adb62c56cad81b112d38a733ef855e
I20260925 11:07:48.840969 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--builtin_ntp_servers=127.8.181.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 11:07:48.854900 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:48.836365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:40933 (local address 127.8.181.62:40365)
0925 11:07:48.837441 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:48.837448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:48.842636 (+ 5188us) server_negotiation.cc:407] Connection header received
0925 11:07:48.843797 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:48.843807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:48.843945 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:48.844105 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:48.846229 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:48.847852 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:48.850954 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:48.851564 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:48.851760 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:48.853701 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:48.853709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:48.853722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:48.853751 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:48.853814 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:48.853831 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:48.853971 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:48.854276 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 11:07:48.854414 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":329,"threads_started":1}
I20260925 11:07:48.856669 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:48.837068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40365 (local address 127.8.181.1:40933)
0925 11:07:48.839725 (+ 2657us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:48.839793 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:48.842701 (+ 2908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:48.844723 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:48.844762 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:48.844942 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:48.845890 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:48.845926 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:48.848105 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:48.848116 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:48.850695 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:48.850721 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:48.851887 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:48.851896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:48.852211 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:48.853104 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:48.853107 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:48.853159 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:48.853279 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:48.853974 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:48.853991 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:48.854003 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 11:07:48.854261 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 11:07:48.854872 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":656,"threads_started":1}
I20260925 11:07:48.860337 18440 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40365
I20260925 11:07:48.860987 18440 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:48.862383 18440 heartbeater.cc:511] Master 127.8.181.62:40365 requested a full tablet report, sending...
I20260925 11:07:48.866767 18254 ts_manager.cc:194] Registered new tserver with Master: 375287d0f98f47669c17d4fb86e0648d (127.8.181.1:39783)
I20260925 11:07:48.869670 18254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.1:40933
W20260925 11:07:49.285182 18444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:49.285540 18444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:49.285643 18444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:49.294603 18444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:49.294751 18444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:49.294869 18444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:49.305430 18444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:49.307008 18444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:49.309288 18444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:49.329948 18452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:49.330592 18450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:49.330545 18449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:49.332046 18444 server_base.cc:1034] running on GCE node
I20260925 11:07:49.333033 18444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:49.334621 18444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:49.338162 18444 hybrid_clock.cc:648] HybridClock initialized: now 1790334469337980 us; error 131 us; skew 500 ppm
I20260925 11:07:49.338917 18444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:49.343686 18444 webserver.cc:460] Webserver started at http://127.8.181.2:36579/ using document root <none> and password file <none>
I20260925 11:07:49.344996 18444 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:49.345153 18444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:49.345557 18444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:49.349133 18444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "770e051e65b3407fad0cb4cbde8db268"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "1b04fb64a217746b98d6968aeda52bf4"
server_key_iv: "384f5499421691458c90b3662d6a119e"
server_key_version: "encryptionkey@0"
I20260925 11:07:49.350622 18444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "770e051e65b3407fad0cb4cbde8db268"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "1b04fb64a217746b98d6968aeda52bf4"
server_key_iv: "384f5499421691458c90b3662d6a119e"
server_key_version: "encryptionkey@0"
I20260925 11:07:49.360642 18444 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260925 11:07:49.369392 18458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:49.372856 18444 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260925 11:07:49.373252 18444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "770e051e65b3407fad0cb4cbde8db268"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "1b04fb64a217746b98d6968aeda52bf4"
server_key_iv: "384f5499421691458c90b3662d6a119e"
server_key_version: "encryptionkey@0"
I20260925 11:07:49.373646 18444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:49.403698 18444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:49.404697 18444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:49.405014 18444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:49.406342 18444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:49.409384 18444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:49.409533 18444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:49.409668 18444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:49.409765 18444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:49.460498 18444 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:38879
I20260925 11:07:49.460624 18570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:38879 every 8 connection(s)
I20260925 11:07:49.462455 18444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:49.464088 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18444
I20260925 11:07:49.464300 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:49.464799 8916 external_mini_cluster.cc:1428] Setting key 312ed14e883d5e41b2fcbca0c78f01de
I20260925 11:07:49.472314 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--builtin_ntp_servers=127.8.181.20:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 11:07:49.484404 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:49.468097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:60813 (local address 127.8.181.62:40365)
0925 11:07:49.468907 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:49.468915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:49.473281 (+ 4366us) server_negotiation.cc:407] Connection header received
0925 11:07:49.475473 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:49.475479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:49.475602 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:49.475743 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:49.477022 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:49.477778 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:49.479869 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:49.480908 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:49.481460 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:49.483420 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:49.483426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:49.483429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:49.483454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:49.483503 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:49.483517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:49.483644 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:49.483902 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 11:07:49.484028 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":520,"threads_started":1}
I20260925 11:07:49.485747 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:49.470893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40365 (local address 127.8.181.2:60813)
0925 11:07:49.473052 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:49.473108 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:49.475073 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:49.476028 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:49.476038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:49.476165 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:49.476821 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:49.476840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:49.477974 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:49.477978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:49.479690 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:49.479707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:49.481562 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:49.481571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:49.481969 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:49.482923 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:49.482929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:49.483003 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:49.483145 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:49.483635 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:49.483646 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:49.483658 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 11:07:49.483875 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 11:07:49.484373 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":807,"threads_started":1}
I20260925 11:07:49.488076 18571 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40365
I20260925 11:07:49.488545 18571 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:49.490370 18571 heartbeater.cc:511] Master 127.8.181.62:40365 requested a full tablet report, sending...
I20260925 11:07:49.493782 18254 ts_manager.cc:194] Registered new tserver with Master: 770e051e65b3407fad0cb4cbde8db268 (127.8.181.2:38879)
I20260925 11:07:49.495398 18254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.2:60813
I20260925 11:07:49.874562 18440 heartbeater.cc:503] Master 127.8.181.62:40365 was elected leader, sending a full tablet report...
W20260925 11:07:49.931879 18575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:49.932214 18575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:49.932343 18575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:49.940750 18575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:49.940907 18575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:49.941044 18575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:49.953011 18575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:49.955044 18575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:49.958093 18575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:49.978295 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:49.978241 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:49.978241 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:49.980888 18575 server_base.cc:1034] running on GCE node
I20260925 11:07:49.982209 18575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:49.983835 18575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:49.987526 18575 hybrid_clock.cc:648] HybridClock initialized: now 1790334469987360 us; error 171 us; skew 500 ppm
I20260925 11:07:49.988160 18575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:49.992069 18575 webserver.cc:460] Webserver started at http://127.8.181.3:45657/ using document root <none> and password file <none>
I20260925 11:07:49.992954 18575 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:49.993084 18575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:49.993407 18575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:49.997035 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "2e906ff0721b425b80ee0e37848bbba3"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "b63a53491c1f508753a357b2f1319355"
server_key_iv: "e6c0dfaf6c7ca12fa7887abc485c6fda"
server_key_version: "encryptionkey@0"
I20260925 11:07:49.998580 18575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "2e906ff0721b425b80ee0e37848bbba3"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "b63a53491c1f508753a357b2f1319355"
server_key_iv: "e6c0dfaf6c7ca12fa7887abc485c6fda"
server_key_version: "encryptionkey@0"
I20260925 11:07:50.008551 18575 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260925 11:07:50.014791 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:50.017105 18575 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260925 11:07:50.017395 18575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "2e906ff0721b425b80ee0e37848bbba3"
format_stamp: "Formatted at 2026-09-25 11:07:49 on dist-test-slave-6042"
server_key: "b63a53491c1f508753a357b2f1319355"
server_key_iv: "e6c0dfaf6c7ca12fa7887abc485c6fda"
server_key_version: "encryptionkey@0"
I20260925 11:07:50.017704 18575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:50.045563 18575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:50.046696 18575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:50.047019 18575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:50.048281 18575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:50.050099 18575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:50.050266 18575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:50.050428 18575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:50.050582 18575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 11:07:50.097184 18575 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45645
I20260925 11:07:50.097481 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45645 every 8 connection(s)
I20260925 11:07:50.099480 18575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:50.107720 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18575
I20260925 11:07:50.107918 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:50.108523 8916 external_mini_cluster.cc:1428] Setting key 9c10796336357aad79897d98db1bb97f
I20260925 11:07:50.121766 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:50.104278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:57911 (local address 127.8.181.62:40365)
0925 11:07:50.105266 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:50.105277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:50.106287 (+ 1010us) server_negotiation.cc:407] Connection header received
0925 11:07:50.109251 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:50.109260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:50.109583 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:50.109863 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:50.113260 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.114532 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.116744 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.117689 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.117899 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:50.120543 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:50.120549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:50.120553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:50.120577 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:50.120638 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:50.120654 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:50.120761 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:50.121225 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 11:07:50.121359 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":662,"threads_started":1}
I20260925 11:07:50.122988 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:50.104837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40365 (local address 127.8.181.3:57911)
0925 11:07:50.106033 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:50.106076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:50.108599 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:50.110673 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:50.110691 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:50.110865 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:50.112684 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:50.112733 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.114829 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.114836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:50.116425 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:50.116451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.118085 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.118092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:50.118457 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:50.119819 (+ 1362us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:50.119823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:50.120072 (+ 249us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:50.120220 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:50.120783 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:50.120792 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:50.120801 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 11:07:50.120989 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 11:07:50.121419 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":401,"threads_started":1}
I20260925 11:07:50.125036 18702 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40365
I20260925 11:07:50.125659 18702 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:50.126912 18702 heartbeater.cc:511] Master 127.8.181.62:40365 requested a full tablet report, sending...
I20260925 11:07:50.130079 18254 ts_manager.cc:194] Registered new tserver with Master: 2e906ff0721b425b80ee0e37848bbba3 (127.8.181.3:45645)
I20260925 11:07:50.131143 18254 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.3:57911
I20260925 11:07:50.140985 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 11:07:50.148486 18704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:50.146605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.8.181.62:40365)
0925 11:07:50.146918 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:50.146926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:50.147326 (+ 400us) server_negotiation.cc:407] Connection header received
0925 11:07:50.147570 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:50.147575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:50.147671 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:50.147795 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:50.148264 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52318: BlockingRecv error: recv got EOF from 127.0.0.1:52318 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 11:07:50.149905 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18313
I20260925 11:07:50.162807 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:50.151762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.8.181.1:39783)
0925 11:07:50.152406 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:50.152414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:50.152447 (+ 33us) server_negotiation.cc:407] Connection header received
0925 11:07:50.152628 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:50.152636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:50.152907 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:50.153094 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:50.153763 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.154987 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.156583 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.157271 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.157408 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:50.159939 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:50.159952 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:50.159969 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:50.161205 (+ 1236us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:50.161282 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:50.161322 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:50.161552 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:50.162043 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 11:07:50.162224 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260925 11:07:50.164317 18435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:50.263489 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18313
I20260925 11:07:50.284742 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18444
I20260925 11:07:50.297432 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:50.286792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.8.181.2:38879)
0925 11:07:50.287386 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:50.287394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:50.287427 (+ 33us) server_negotiation.cc:407] Connection header received
0925 11:07:50.287499 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:50.287505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:50.287754 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:50.287924 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:50.288468 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.289748 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.291727 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.292504 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.292649 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:50.294302 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:50.294316 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:50.294337 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:50.295382 (+ 1045us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:50.295641 (+ 259us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:50.295693 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:50.295847 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:50.296522 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 11:07:50.296865 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":306,"threads_started":1}
I20260925 11:07:50.299546 18566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:50.407207 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18444
I20260925 11:07:50.427753 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18575
I20260925 11:07:50.439090 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:50.429022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.8.181.3:45645)
0925 11:07:50.430370 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:50.430383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:50.430433 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:07:50.430661 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:50.430671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:50.431003 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:50.431245 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:50.432030 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.433167 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.434772 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:50.435405 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:50.435568 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:50.437060 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:50.437071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:50.437091 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:50.437814 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:50.437874 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:50.437906 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:50.438065 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:50.438333 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 11:07:50.438571 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":552,"threads_started":1}
I20260925 11:07:50.440621 18697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:50.540024 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18575
I20260925 11:07:50.560073 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18223
I20260925 11:07:50.562063 18284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:50.686376 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18223
2026-09-25T11:07:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3385 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T11:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:50Z Disabled control of system clock
I20260925 11:07:50.780441 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:33705
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:33705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 11:07:51.202579 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:51.203150 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:51.203372 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:51.212770 18722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:51.212930 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:51.213017 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:51.213129 18722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:51.213219 18722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:51.223652 18722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:33705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:33705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:51.225219 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:51.228142 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:51.245658 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:51.245523 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:51.245523 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:51.247484 18722 server_base.cc:1034] running on GCE node
I20260925 11:07:51.248667 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:51.250694 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:51.252202 18722 hybrid_clock.cc:648] HybridClock initialized: now 1790334471252093 us; error 114 us; skew 500 ppm
I20260925 11:07:51.252794 18722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:51.256325 18722 webserver.cc:460] Webserver started at http://127.8.181.62:34915/ using document root <none> and password file <none>
I20260925 11:07:51.257618 18722 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:51.257815 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:51.258286 18722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:51.264678 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "6fd83038fcb04f7abd37f16df4d449d8"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8ee0f34529d64efcffa78b99dc4dde23"
server_key_iv: "5ada0ad07299393835c7374e6fa8a4e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:51.267158 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "6fd83038fcb04f7abd37f16df4d449d8"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8ee0f34529d64efcffa78b99dc4dde23"
server_key_iv: "5ada0ad07299393835c7374e6fa8a4e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:51.277408 18722 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260925 11:07:51.286005 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:51.288882 18722 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 11:07:51.289194 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "6fd83038fcb04f7abd37f16df4d449d8"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8ee0f34529d64efcffa78b99dc4dde23"
server_key_iv: "5ada0ad07299393835c7374e6fa8a4e7"
server_key_version: "encryptionkey@0"
I20260925 11:07:51.289475 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:51.296901 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:51.297931 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:51.298257 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:51.328517 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:33705 every 8 connection(s)
I20260925 11:07:51.328680 18722 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:33705
I20260925 11:07:51.331269 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:51.333709 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18722
I20260925 11:07:51.333998 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:51.334797 8916 external_mini_cluster.cc:1428] Setting key a4cad96f03fc64d6d58da1b3f667f409
I20260925 11:07:51.338145 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:51.350632 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:51.337557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.8.181.62:33705)
0925 11:07:51.341064 (+ 3507us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:51.341085 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:51.341166 (+ 81us) server_negotiation.cc:407] Connection header received
0925 11:07:51.344009 (+ 2843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:51.344045 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:51.344904 (+ 859us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:51.345525 (+ 621us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:51.347249 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:51.347259 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:51.347283 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:51.347309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:51.347428 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:51.347466 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:51.348012 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:51.348217 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 11:07:51.348835 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1879,"thread_start_us":1322,"threads_started":1}
I20260925 11:07:51.358531 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8: Bootstrap starting.
I20260925 11:07:51.362784 18788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:51.364003 18788 log.cc:824] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:51.368052 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8: No bootstrap required, opened a new log
I20260925 11:07:51.378187 18788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } }
I20260925 11:07:51.379262 18788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:51.379428 18788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd83038fcb04f7abd37f16df4d449d8, State: Initialized, Role: FOLLOWER
I20260925 11:07:51.381209 18788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } }
I20260925 11:07:51.383174 18788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:51.383693 18788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:51.384253 18788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:51.387830 18788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } }
I20260925 11:07:51.388581 18788 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd83038fcb04f7abd37f16df4d449d8; no voters:
I20260925 11:07:51.389520 18788 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:51.389642 18793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:51.390532 18793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd83038fcb04f7abd37f16df4d449d8, State: Running, Role: LEADER
I20260925 11:07:51.391439 18793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } }
I20260925 11:07:51.392516 18788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:51.396289 18795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd83038fcb04f7abd37f16df4d449d8. Latest consensus state: current_term: 1 leader_uuid: "6fd83038fcb04f7abd37f16df4d449d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } } }
I20260925 11:07:51.396296 18794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd83038fcb04f7abd37f16df4d449d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd83038fcb04f7abd37f16df4d449d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 33705 } } }
I20260925 11:07:51.396915 18795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:51.396957 18794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:51.403573 18799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:51.413110 18799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:51.426017 18799 catalog_manager.cc:1347] Generated new cluster ID: b0bd5f515c3547b4ad944ee7afa75302
I20260925 11:07:51.426215 18799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:51.449167 18799 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:51.450163 18799 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:51.457681 18799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd83038fcb04f7abd37f16df4d449d8: Generated new TSK 0
I20260925 11:07:51.459492 18799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:07:51.478179 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--builtin_ntp_servers=127.8.181.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 11:07:51.942229 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:51.942714 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:51.942952 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:51.951846 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:51.951990 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:51.952194 18812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:51.963891 18812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:51.965425 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:51.967923 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:51.986160 18818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:51.986239 18820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:51.986446 18817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:51.986966 18812 server_base.cc:1034] running on GCE node
I20260925 11:07:51.988147 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:51.989535 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:51.991176 18812 hybrid_clock.cc:648] HybridClock initialized: now 1790334471991065 us; error 90 us; skew 500 ppm
I20260925 11:07:51.991854 18812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:51.995579 18812 webserver.cc:460] Webserver started at http://127.8.181.1:43169/ using document root <none> and password file <none>
I20260925 11:07:51.997288 18812 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:51.997756 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:51.998584 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:52.002183 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "03c5c661bfb946b39f2892c9699a0596"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8104a303b062db1ec935d2c3b11f38e9"
server_key_iv: "22bfd0ee5e41db76444d32f7cc986440"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.003273 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "03c5c661bfb946b39f2892c9699a0596"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8104a303b062db1ec935d2c3b11f38e9"
server_key_iv: "22bfd0ee5e41db76444d32f7cc986440"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.011166 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260925 11:07:52.017891 18826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.021293 18812 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 11:07:52.021654 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "03c5c661bfb946b39f2892c9699a0596"
format_stamp: "Formatted at 2026-09-25 11:07:51 on dist-test-slave-6042"
server_key: "8104a303b062db1ec935d2c3b11f38e9"
server_key_iv: "22bfd0ee5e41db76444d32f7cc986440"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.021997 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:52.029984 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:52.031152 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:52.031533 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:52.032847 18812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:52.034833 18812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:52.035012 18812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.035185 18812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:52.035291 18812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.086462 18812 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:44511
I20260925 11:07:52.086549 18938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:44511 every 8 connection(s)
I20260925 11:07:52.088714 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:52.096961 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18812
I20260925 11:07:52.097226 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:52.098096 8916 external_mini_cluster.cc:1428] Setting key ab2e89299a48f134e31ff8e99b3512c3
I20260925 11:07:52.102041 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--builtin_ntp_servers=127.8.181.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 11:07:52.103011 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:52.093406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:38301 (local address 127.8.181.62:33705)
0925 11:07:52.094380 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:52.094386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:52.096077 (+ 1691us) server_negotiation.cc:407] Connection header received
0925 11:07:52.098987 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:52.098993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:52.099089 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:52.099223 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:52.101688 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:52.101695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:52.101699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:52.101730 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:52.101804 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:52.101822 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:52.102091 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:52.102570 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 11:07:52.102662 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":573,"threads_started":1}
I20260925 11:07:52.104612 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:52.094088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:33705 (local address 127.8.181.1:38301)
0925 11:07:52.095848 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:52.095905 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:52.098521 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:52.099648 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:52.099664 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:52.099828 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:52.100837 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:52.100840 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:52.101015 (+ 175us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:52.101180 (+ 165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:52.102174 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:52.102204 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:52.102216 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 11:07:52.102615 (+ 399us) client_negotiation.cc:240] Negotiation successful
0925 11:07:52.102984 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":670,"threads_started":1}
I20260925 11:07:52.107165 18939 heartbeater.cc:347] Connected to a master server at 127.8.181.62:33705
I20260925 11:07:52.107859 18939 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:52.109177 18939 heartbeater.cc:511] Master 127.8.181.62:33705 requested a full tablet report, sending...
I20260925 11:07:52.112761 18753 ts_manager.cc:194] Registered new tserver with Master: 03c5c661bfb946b39f2892c9699a0596 (127.8.181.1:44511)
W20260925 11:07:52.502460 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:52.502942 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:52.503063 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:52.512365 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:52.512517 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:52.512642 18943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:52.523952 18943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:52.526027 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:52.530620 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:52.549732 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:52.549757 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:52.549682 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:52.550603 18943 server_base.cc:1034] running on GCE node
I20260925 11:07:52.551725 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:52.553783 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:52.555122 18943 hybrid_clock.cc:648] HybridClock initialized: now 1790334472555065 us; error 99 us; skew 500 ppm
I20260925 11:07:52.555541 18943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:52.558947 18943 webserver.cc:460] Webserver started at http://127.8.181.2:46071/ using document root <none> and password file <none>
I20260925 11:07:52.559823 18943 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:52.559979 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:52.560346 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:52.563767 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "c838587d5d0e4033a13426b76293bb75"
format_stamp: "Formatted at 2026-09-25 11:07:52 on dist-test-slave-6042"
server_key: "50e0a344919cde855f4313f4f98fc989"
server_key_iv: "e71e1d23bfc9cd6de81a6d362673229b"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.564755 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "c838587d5d0e4033a13426b76293bb75"
format_stamp: "Formatted at 2026-09-25 11:07:52 on dist-test-slave-6042"
server_key: "50e0a344919cde855f4313f4f98fc989"
server_key_iv: "e71e1d23bfc9cd6de81a6d362673229b"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.572005 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260925 11:07:52.576845 18957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.579479 18943 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 11:07:52.579692 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "c838587d5d0e4033a13426b76293bb75"
format_stamp: "Formatted at 2026-09-25 11:07:52 on dist-test-slave-6042"
server_key: "50e0a344919cde855f4313f4f98fc989"
server_key_iv: "e71e1d23bfc9cd6de81a6d362673229b"
server_key_version: "encryptionkey@0"
I20260925 11:07:52.579979 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:52.589550 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:52.591741 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:52.592185 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:52.593807 18943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:52.596421 18943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:52.596545 18943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.596648 18943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:52.596769 18943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:52.643273 18943 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:33643
I20260925 11:07:52.643396 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:33643 every 8 connection(s)
I20260925 11:07:52.644806 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:52.652835 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 18943
I20260925 11:07:52.653062 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:52.653644 8916 external_mini_cluster.cc:1428] Setting key 7aca896ebbb6f4af756939ded3a5e3a3
I20260925 11:07:52.657094 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:52.649076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:38969 (local address 127.8.181.62:33705)
0925 11:07:52.649877 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:52.649888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:52.651346 (+ 1458us) server_negotiation.cc:407] Connection header received
0925 11:07:52.653982 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:52.653988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:52.654059 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:52.654175 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:52.656034 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:52.656043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:52.656057 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:52.656084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:52.656137 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:52.656162 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:52.656407 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:52.656658 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 11:07:52.656747 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":436,"threads_started":1}
I20260925 11:07:52.657581 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--builtin_ntp_servers=127.8.181.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 11:07:52.658595 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:52.649392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:33705 (local address 127.8.181.2:38969)
0925 11:07:52.651082 (+ 1690us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:52.651131 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:52.653455 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:52.654567 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:52.654579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:52.654710 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:52.655548 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:52.655552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:52.655615 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:52.655766 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:52.656375 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:52.656387 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:52.656395 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 11:07:52.656858 (+ 463us) client_negotiation.cc:240] Negotiation successful
0925 11:07:52.657194 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":467,"threads_started":1}
I20260925 11:07:52.660696 19070 heartbeater.cc:347] Connected to a master server at 127.8.181.62:33705
I20260925 11:07:52.661101 19070 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:52.661844 19070 heartbeater.cc:511] Master 127.8.181.62:33705 requested a full tablet report, sending...
I20260925 11:07:52.663856 18753 ts_manager.cc:194] Registered new tserver with Master: c838587d5d0e4033a13426b76293bb75 (127.8.181.2:33643)
W20260925 11:07:53.049074 19074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:53.049472 19074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:53.049649 19074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:53.057494 19074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:53.057677 19074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:53.057876 19074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:53.067930 19074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:33705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:53.069350 19074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:53.071381 19074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:53.088126 19079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:53.088507 19080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:53.088995 19082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:53.092950 19074 server_base.cc:1034] running on GCE node
I20260925 11:07:53.093817 19074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:53.095331 19074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:53.097046 19074 hybrid_clock.cc:648] HybridClock initialized: now 1790334473096919 us; error 143 us; skew 500 ppm
I20260925 11:07:53.097556 19074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:07:53.100494 19074 webserver.cc:460] Webserver started at http://127.8.181.3:33785/ using document root <none> and password file <none>
I20260925 11:07:53.101320 19074 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:53.101469 19074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:53.101864 19074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:53.105604 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "d8f10c5625384ef39f4b961aa42c3ebf"
format_stamp: "Formatted at 2026-09-25 11:07:53 on dist-test-slave-6042"
server_key: "118417e3d5bb0355f82fe7ff64eafde1"
server_key_iv: "c2d7c581ad93b78d3d3b98a5aeb11ce2"
server_key_version: "encryptionkey@0"
I20260925 11:07:53.106622 19074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f10c5625384ef39f4b961aa42c3ebf"
format_stamp: "Formatted at 2026-09-25 11:07:53 on dist-test-slave-6042"
server_key: "118417e3d5bb0355f82fe7ff64eafde1"
server_key_iv: "c2d7c581ad93b78d3d3b98a5aeb11ce2"
server_key_version: "encryptionkey@0"
I20260925 11:07:53.113893 19074 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 11:07:53.116966 18939 heartbeater.cc:503] Master 127.8.181.62:33705 was elected leader, sending a full tablet report...
I20260925 11:07:53.118983 19088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:53.121367 19074 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:07:53.121698 19074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "d8f10c5625384ef39f4b961aa42c3ebf"
format_stamp: "Formatted at 2026-09-25 11:07:53 on dist-test-slave-6042"
server_key: "118417e3d5bb0355f82fe7ff64eafde1"
server_key_iv: "c2d7c581ad93b78d3d3b98a5aeb11ce2"
server_key_version: "encryptionkey@0"
I20260925 11:07:53.121989 19074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:53.128894 19074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:53.129904 19074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:53.130227 19074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:53.131438 19074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:53.133131 19074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:53.133270 19074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:53.133410 19074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:53.133498 19074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:53.171640 19074 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:37449
I20260925 11:07:53.171891 19200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:37449 every 8 connection(s)
I20260925 11:07:53.173159 19074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:53.180202 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 19074
I20260925 11:07:53.180416 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:53.181007 8916 external_mini_cluster.cc:1428] Setting key 3bae3dc9ff91297fd205cdd54ec0d7cb
I20260925 11:07:53.186847 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:53.177276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:42863 (local address 127.8.181.62:33705)
0925 11:07:53.178062 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:53.178067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:53.180215 (+ 2148us) server_negotiation.cc:407] Connection header received
0925 11:07:53.182962 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:53.183019 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:53.183115 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:53.183393 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:53.185710 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:53.185724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:53.185727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:53.185751 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:53.185805 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:53.185819 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:53.186039 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:53.186411 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 11:07:53.186540 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":485,"threads_started":1}
I20260925 11:07:53.188040 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:53.178311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:33705 (local address 127.8.181.3:42863)
0925 11:07:53.179910 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:53.179987 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:53.182487 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:53.184132 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:53.184144 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:53.184307 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:53.185308 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:07:53.185311 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:53.185363 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:07:53.185472 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:53.186049 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:53.186061 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:53.186138 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 11:07:53.186452 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 11:07:53.186806 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":508,"threads_started":1}
I20260925 11:07:53.189766 19201 heartbeater.cc:347] Connected to a master server at 127.8.181.62:33705
I20260925 11:07:53.190096 19201 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:53.190979 19201 heartbeater.cc:511] Master 127.8.181.62:33705 requested a full tablet report, sending...
I20260925 11:07:53.192848 18753 ts_manager.cc:194] Registered new tserver with Master: d8f10c5625384ef39f4b961aa42c3ebf (127.8.181.3:37449)
I20260925 11:07:53.198588 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 11:07:53.206156 19203 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:07:53.204322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.8.181.62:33705)
0925 11:07:53.204613 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:53.204620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:53.205009 (+ 389us) server_negotiation.cc:407] Connection header received
0925 11:07:53.205279 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:53.205284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:53.205357 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:53.205476 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:53.205930 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55690: BlockingRecv error: recv got EOF from 127.0.0.1:55690 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 11:07:53.207490 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18812
I20260925 11:07:53.212464 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:53.209242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.8.181.1:44511)
0925 11:07:53.209915 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:53.209923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:53.209950 (+ 27us) server_negotiation.cc:407] Connection header received
0925 11:07:53.210010 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:53.210016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:53.210241 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:53.210423 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:53.211687 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:53.211694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:53.211706 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:53.211718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:53.211760 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:53.211774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:53.211967 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:53.212036 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 11:07:53.212145 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":369,"threads_started":1}
I20260925 11:07:53.213913 18934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:53.328877 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18812
I20260925 11:07:53.351161 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18943
I20260925 11:07:53.355399 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:53.352340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.8.181.2:33643)
0925 11:07:53.352844 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:53.352852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:53.352880 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:07:53.352941 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:53.352948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:53.353174 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:53.353391 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:53.354452 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:53.354458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:53.354520 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:53.354546 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:53.354599 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:53.354621 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:53.354776 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:53.355004 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 11:07:53.355110 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":227,"threads_started":1}
I20260925 11:07:53.356841 19065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:53.445775 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18943
I20260925 11:07:53.463881 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 19074
I20260925 11:07:53.468713 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:53.465082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.8.181.3:37449)
0925 11:07:53.465725 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:53.465733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:53.465765 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:07:53.465829 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:53.465836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:53.466070 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:53.466447 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:53.467892 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:53.467899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:53.467912 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:07:53.467927 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:53.467973 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:07:53.467992 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:53.468241 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:53.468311 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 11:07:53.468422 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":306,"threads_started":1}
I20260925 11:07:53.470329 19196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:53.559712 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 19074
I20260925 11:07:53.576373 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 18722
I20260925 11:07:53.577962 18783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:53.684222 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 18722
2026-09-25T11:07:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2991 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:53 dist-test-slave-6042 krb5kdc[19220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:53 dist-test-slave-6042 krb5kdc[19220](info): set up 2 sockets
Sep 25 11:07:53 dist-test-slave-6042 krb5kdc[19220](info): commencing operation
krb5kdc: starting...
W20260925 11:07:54.864319 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:55 dist-test-slave-6042 krb5kdc[19220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:07:55.132472 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32889
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:55.499015 19233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:55.499353 19233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:55.499449 19233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:55.507166 19233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:07:55.507297 19233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:55.507344 19233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:55.507393 19233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:07:55.507499 19233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:07:55.518098 19233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:55.519562 19233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:55.520906 19233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:55.534364 19241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:55.534441 19239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:55.534564 19238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:55.535238 19233 server_base.cc:1034] running on GCE node
I20260925 11:07:55.536166 19233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:55.537854 19233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:55.539108 19233 hybrid_clock.cc:648] HybridClock initialized: now 1790334475539030 us; error 73 us; skew 500 ppm
I20260925 11:07:55.539482 19233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:55 dist-test-slave-6042 krb5kdc[19220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:55.543066 19233 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:07:55.544862 19233 webserver.cc:460] Webserver started at http://127.8.181.62:34973/ using document root <none> and password file <none>
I20260925 11:07:55.545635 19233 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:55.545799 19233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:55.546118 19233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:55.549557 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "823ad5a6436143799a053577439bad41"
format_stamp: "Formatted at 2026-09-25 11:07:55 on dist-test-slave-6042"
server_key: "db83bd9cf8eecf4e8587039fc17d8f99"
server_key_iv: "03c56f6129b253ae7aa34ebb761b6780"
server_key_version: "encryptionkey@0"
I20260925 11:07:55.550411 19233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "823ad5a6436143799a053577439bad41"
format_stamp: "Formatted at 2026-09-25 11:07:55 on dist-test-slave-6042"
server_key: "db83bd9cf8eecf4e8587039fc17d8f99"
server_key_iv: "03c56f6129b253ae7aa34ebb761b6780"
server_key_version: "encryptionkey@0"
I20260925 11:07:55.556488 19233 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 11:07:55.560904 19248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:55.562712 19233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:07:55.562939 19233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "823ad5a6436143799a053577439bad41"
format_stamp: "Formatted at 2026-09-25 11:07:55 on dist-test-slave-6042"
server_key: "db83bd9cf8eecf4e8587039fc17d8f99"
server_key_iv: "03c56f6129b253ae7aa34ebb761b6780"
server_key_version: "encryptionkey@0"
I20260925 11:07:55.563194 19233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:55.591384 19233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:55.596616 19233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:55.596958 19233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:55.617985 19233 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:32889
I20260925 11:07:55.618026 19299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:32889 every 8 connection(s)
I20260925 11:07:55.619655 19233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:07:55.620234 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 19233
I20260925 11:07:55.620491 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:07:55.621371 8916 external_mini_cluster.cc:1428] Setting key f1a997b6d2c4e564afad29b5eb57a5b3
I20260925 11:07:55.624814 19300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:55.637270 19300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: Bootstrap starting.
Sep 25 11:07:55 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:55.641079 19300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:55.642292 19300 log.cc:824] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:55.646091 19300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: No bootstrap required, opened a new log
I20260925 11:07:55.647517 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:55.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.8.181.62:32889)
0925 11:07:55.624785 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:55.624802 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:55.624848 (+ 46us) server_negotiation.cc:407] Connection header received
0925 11:07:55.627089 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:55.627122 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:55.627731 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:55.628510 (+ 779us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:55.630368 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:55.631989 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:55.633435 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:55.634242 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:55.634466 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:55.639157 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:55.639167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:55.639183 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:55.639237 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:55.642791 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:55.643678 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:55.643687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:55.643698 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:55.643888 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:55.644417 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:55.644421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:55.644423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:55.645051 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:55.645303 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:55.645931 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 11:07:55.646407 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":491,"threads_started":1}
I20260925 11:07:55.655315 19300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } }
I20260925 11:07:55.655838 19300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:55.655961 19300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823ad5a6436143799a053577439bad41, State: Initialized, Role: FOLLOWER
I20260925 11:07:55.656809 19300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } }
I20260925 11:07:55.657125 19300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:07:55.657290 19300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:07:55.657508 19300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:55.660339 19300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } }
I20260925 11:07:55.660959 19300 leader_election.cc:302] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 823ad5a6436143799a053577439bad41; no voters:
I20260925 11:07:55.661674 19300 leader_election.cc:288] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:07:55.661875 19305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:55.663206 19305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [term 1 LEADER]: Becoming Leader. State: Replica: 823ad5a6436143799a053577439bad41, State: Running, Role: LEADER
I20260925 11:07:55.664206 19300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:07:55.664085 19305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } }
I20260925 11:07:55.667598 19306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "823ad5a6436143799a053577439bad41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } } }
I20260925 11:07:55.668063 19306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:55.667573 19307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 823ad5a6436143799a053577439bad41. Latest consensus state: current_term: 1 leader_uuid: "823ad5a6436143799a053577439bad41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "823ad5a6436143799a053577439bad41" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 32889 } } }
I20260925 11:07:55.668708 19307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41 [sys.catalog]: This master's current role is: LEADER
I20260925 11:07:55.670445 19314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:07:55.675173 19314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:07:55.684432 19314 catalog_manager.cc:1347] Generated new cluster ID: 15ada9115a1a4f609ecc40317353503b
I20260925 11:07:55.684574 19314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:07:55.723614 19314 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:07:55.725234 19314 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:07:55.735539 19314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: Generated new TSK 0
I20260925 11:07:55.736532 19314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:07:55.807076 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:56.168151 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:56.168511 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:56.168684 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:56.177074 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:56.177220 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:56.177403 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:07:56.188807 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:56.190346 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:56.192257 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:56.207930 19332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:56.207836 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:56.207886 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:56.209362 19326 server_base.cc:1034] running on GCE node
I20260925 11:07:56.210196 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:56.211416 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:56.212632 19326 hybrid_clock.cc:648] HybridClock initialized: now 1790334476212569 us; error 56 us; skew 500 ppm
I20260925 11:07:56.213027 19326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:56 dist-test-slave-6042 krb5kdc[19220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:56.217219 19326 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:07:56.223966 19326 webserver.cc:460] Webserver started at http://127.8.181.1:36931/ using document root <none> and password file <none>
I20260925 11:07:56.224762 19326 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:56.224876 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:56.225132 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:56.228325 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "205bd55b62e74f16a405f36a5e2a9f12"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "4a71dc3c439768e97d67e5cf64891024"
server_key_iv: "ab2ae1920b610f37d3dda0f2f5541691"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.229094 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "205bd55b62e74f16a405f36a5e2a9f12"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "4a71dc3c439768e97d67e5cf64891024"
server_key_iv: "ab2ae1920b610f37d3dda0f2f5541691"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.234942 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:07:56.239069 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:56.240761 19326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:07:56.240978 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "205bd55b62e74f16a405f36a5e2a9f12"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "4a71dc3c439768e97d67e5cf64891024"
server_key_iv: "ab2ae1920b610f37d3dda0f2f5541691"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.241220 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:56.266832 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:56.272284 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:56.272632 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:56.274082 19326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:56.275947 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:56.276229 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:56.276418 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:56.276496 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:56.312969 19326 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:44725
I20260925 11:07:56.313030 19453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:44725 every 8 connection(s)
I20260925 11:07:56.314324 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:07:56.324198 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 19326
I20260925 11:07:56.324463 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:07:56.325103 8916 external_mini_cluster.cc:1428] Setting key 605bf61669bd42c3574dcfe54ea33a0e
Sep 25 11:07:56 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:56.337466 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:56.317204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:50003 (local address 127.8.181.62:32889)
0925 11:07:56.317666 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:56.317671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:56.318818 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 11:07:56.320650 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:56.320655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:56.320729 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:56.320835 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:56.322410 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.323245 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.325256 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.325667 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.325818 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:56.330598 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:56.330604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:56.330608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:56.330646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:56.333470 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:56.334705 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:56.334713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:56.334719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:56.334880 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:56.335495 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:56.335500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:56.335504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:56.335799 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:56.335906 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:56.336841 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 11:07:56.337089 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260925 11:07:56.338603 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:56.317802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32889 (local address 127.8.181.1:50003)
0925 11:07:56.318585 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:56.318635 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:56.320335 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:56.321060 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:56.321068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:56.321598 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:56.322193 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:56.322210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.323450 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.323454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:56.325063 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:56.325080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.325939 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.325944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:56.326275 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:56.327151 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:56.327184 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:56.330245 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:56.333669 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:56.333683 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:56.333706 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:56.334424 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:56.335022 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:56.335029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:56.335036 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:56.335332 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:56.335987 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:56.335993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:56.336404 (+ 411us) client_negotiation.cc:769] Sending connection context
0925 11:07:56.336710 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 11:07:56.337293 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":273,"threads_started":1}
I20260925 11:07:56.340596 19454 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32889
I20260925 11:07:56.341097 19454 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:56.342123 19454 heartbeater.cc:511] Master 127.8.181.62:32889 requested a full tablet report, sending...
I20260925 11:07:56.344991 19265 ts_manager.cc:194] Registered new tserver with Master: 205bd55b62e74f16a405f36a5e2a9f12 (127.8.181.1:44725)
I20260925 11:07:56.347376 19265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:50003
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:07:56.391404 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:07:56.762857 19460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:56.763237 19460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:56.763410 19460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:56.771579 19460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:56.771752 19460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:56.771945 19460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:07:56.783164 19460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:56.784687 19460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:56.786254 19460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:56.801575 19465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:56.801606 19466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:56.801575 19468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:56.802973 19460 server_base.cc:1034] running on GCE node
I20260925 11:07:56.803755 19460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:56.804956 19460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:56.806262 19460 hybrid_clock.cc:648] HybridClock initialized: now 1790334476806241 us; error 61 us; skew 500 ppm
I20260925 11:07:56.806838 19460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:56 dist-test-slave-6042 krb5kdc[19220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:56.810606 19460 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:07:56.812410 19460 webserver.cc:460] Webserver started at http://127.8.181.2:41255/ using document root <none> and password file <none>
I20260925 11:07:56.813194 19460 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:56.813359 19460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:56.813699 19460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:56.817121 19460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "a0b2e75c2dc04f168f12e62ecbd51072"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "243c18c8328d665f57a76912102c58a4"
server_key_iv: "40d6d4ef3f9efb4f3a9a4524cf162967"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.818049 19460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "a0b2e75c2dc04f168f12e62ecbd51072"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "243c18c8328d665f57a76912102c58a4"
server_key_iv: "40d6d4ef3f9efb4f3a9a4524cf162967"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.824316 19460 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 11:07:56.829200 19475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:56.831821 19460 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 11:07:56.832101 19460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "a0b2e75c2dc04f168f12e62ecbd51072"
format_stamp: "Formatted at 2026-09-25 11:07:56 on dist-test-slave-6042"
server_key: "243c18c8328d665f57a76912102c58a4"
server_key_iv: "40d6d4ef3f9efb4f3a9a4524cf162967"
server_key_version: "encryptionkey@0"
I20260925 11:07:56.832366 19460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:56.855074 19460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:56.859952 19460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:56.860287 19460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:56.861433 19460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:56.863034 19460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:56.863165 19460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:56.863348 19460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:56.863451 19460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 11:07:56.902848 19460 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44599
I20260925 11:07:56.902906 19587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44599 every 8 connection(s)
I20260925 11:07:56.904213 19460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:07:56.909132 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 19460
I20260925 11:07:56.909348 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:07:56.909824 8916 external_mini_cluster.cc:1428] Setting key 0e1632e218a74c757d8d43383a06728e
Sep 25 11:07:56 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:56.928166 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:56.907419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:37831 (local address 127.8.181.62:32889)
0925 11:07:56.908006 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:56.908011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:56.908886 (+ 875us) server_negotiation.cc:407] Connection header received
0925 11:07:56.911199 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:56.911204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:56.911269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:56.911366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:56.913294 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.914164 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.915976 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.916419 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.916567 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:56.921145 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:56.921161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:56.921165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:56.921206 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:56.924066 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:56.925271 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:56.925276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:56.925279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:56.925441 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:56.926039 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:56.926043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:56.926045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:56.926356 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:56.926559 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:56.927554 (+ 995us) server_negotiation.cc:299] Negotiation successful
0925 11:07:56.927776 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":464,"thread_start_us":369,"threads_started":1}
I20260925 11:07:56.929663 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:56.907872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32889 (local address 127.8.181.2:37831)
0925 11:07:56.908699 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:56.908746 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:56.910837 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:56.911689 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:56.911702 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:56.912433 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:56.913095 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:56.913114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.914380 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.914384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:56.915797 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:56.915811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:56.916685 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:56.916692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:56.916940 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:56.917731 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:56.917760 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:56.920769 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:56.924273 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:56.924288 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:56.924314 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:56.925049 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:56.925585 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:56.925594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:56.925600 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:56.925895 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:56.926515 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:56.926534 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:56.927073 (+ 539us) client_negotiation.cc:769] Sending connection context
0925 11:07:56.927419 (+ 346us) client_negotiation.cc:240] Negotiation successful
0925 11:07:56.928063 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":351,"threads_started":1}
I20260925 11:07:56.931891 19588 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32889
I20260925 11:07:56.932307 19588 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:56.933220 19588 heartbeater.cc:511] Master 127.8.181.62:32889 requested a full tablet report, sending...
I20260925 11:07:56.935310 19265 ts_manager.cc:194] Registered new tserver with Master: a0b2e75c2dc04f168f12e62ecbd51072 (127.8.181.2:44599)
I20260925 11:07:56.936290 19265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:37831
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:07:56.976792 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:07:57.352247 19454 heartbeater.cc:503] Master 127.8.181.62:32889 was elected leader, sending a full tablet report...
W20260925 11:07:57.362924 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:07:57.363371 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:07:57.363498 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:07:57.373739 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:07:57.373898 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:07:57.374035 19594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:07:57.385824 19594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:07:57.387270 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:07:57.388657 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:07:57.409111 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:57.409721 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:07:57.409682 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:07:57.410303 19594 server_base.cc:1034] running on GCE node
I20260925 11:07:57.411166 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:07:57.412406 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:07:57.413719 19594 hybrid_clock.cc:648] HybridClock initialized: now 1790334477413653 us; error 87 us; skew 500 ppm
I20260925 11:07:57.414134 19594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:07:57 dist-test-slave-6042 krb5kdc[19220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:07:57.418140 19594 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:07:57.420477 19594 webserver.cc:460] Webserver started at http://127.8.181.3:46461/ using document root <none> and password file <none>
I20260925 11:07:57.421321 19594 fs_manager.cc:359] Metadata directory not provided
I20260925 11:07:57.421494 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:07:57.421844 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:07:57.425711 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "a4f3af3026c34226ba98ca9c73a99d03"
format_stamp: "Formatted at 2026-09-25 11:07:57 on dist-test-slave-6042"
server_key: "05352fd5df6feffa000a2de01bf9de36"
server_key_iv: "a87d3cec59bb3468e64752252c369c60"
server_key_version: "encryptionkey@0"
I20260925 11:07:57.426821 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "a4f3af3026c34226ba98ca9c73a99d03"
format_stamp: "Formatted at 2026-09-25 11:07:57 on dist-test-slave-6042"
server_key: "05352fd5df6feffa000a2de01bf9de36"
server_key_iv: "a87d3cec59bb3468e64752252c369c60"
server_key_version: "encryptionkey@0"
I20260925 11:07:57.434193 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 11:07:57.439205 19609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:57.441377 19594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260925 11:07:57.441766 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "a4f3af3026c34226ba98ca9c73a99d03"
format_stamp: "Formatted at 2026-09-25 11:07:57 on dist-test-slave-6042"
server_key: "05352fd5df6feffa000a2de01bf9de36"
server_key_iv: "a87d3cec59bb3468e64752252c369c60"
server_key_version: "encryptionkey@0"
I20260925 11:07:57.442054 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:07:57.455624 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:07:57.461534 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:07:57.461913 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:07:57.463186 19594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:07:57.465003 19594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:07:57.465137 19594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:57.465286 19594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:07:57.465359 19594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:07:57.506793 19594 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:35659
I20260925 11:07:57.506850 19721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:35659 every 8 connection(s)
I20260925 11:07:57.508220 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:07:57.517722 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 19594
I20260925 11:07:57.517976 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:07:57.518437 8916 external_mini_cluster.cc:1428] Setting key 2f1f05fff545c5d02a2007ca31d3f41c
Sep 25 11:07:57 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:07:57.534566 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.511466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52395 (local address 127.8.181.62:32889)
0925 11:07:57.512324 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.512343 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.513347 (+ 1004us) server_negotiation.cc:407] Connection header received
0925 11:07:57.515886 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.515894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.516024 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.516158 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:57.518632 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.519704 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.522662 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.523408 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.523569 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.527861 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:57.527867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:57.527870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:57.527909 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:57.531067 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.531996 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.532001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.532005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.532141 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.532584 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.532593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.532596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.532890 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:57.533055 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.533880 (+ 825us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.534133 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":422,"threads_started":1}
I20260925 11:07:57.535759 19725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.512185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:32889 (local address 127.8.181.3:52395)
0925 11:07:57.513113 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.513164 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.515333 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.516646 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.516660 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.517394 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:07:57.518298 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.518325 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.519936 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.519942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.522202 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.522232 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.523677 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.523684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.523945 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.524742 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:07:57.524767 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:07:57.527549 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:07:57.531304 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:57.531316 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:57.531337 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:57.531852 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:57.532266 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:07:57.532271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:07:57.532275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:07:57.532473 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:07:57.533031 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:07:57.533066 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:07:57.533436 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.533778 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.534369 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":359,"threads_started":1}
I20260925 11:07:57.537294 19722 heartbeater.cc:347] Connected to a master server at 127.8.181.62:32889
I20260925 11:07:57.537670 19722 heartbeater.cc:464] Registering TS with master...
I20260925 11:07:57.538383 19722 heartbeater.cc:511] Master 127.8.181.62:32889 requested a full tablet report, sending...
I20260925 11:07:57.540218 19265 ts_manager.cc:194] Registered new tserver with Master: a4f3af3026c34226ba98ca9c73a99d03 (127.8.181.3:35659)
I20260925 11:07:57.541146 19265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:52395
I20260925 11:07:57.549335 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:07:57.573180 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.554963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.8.181.62:32889)
0925 11:07:57.555232 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.555237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.555914 (+ 677us) server_negotiation.cc:407] Connection header received
0925 11:07:57.556110 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.556115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.556250 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.556357 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:57.558753 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.559874 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.561508 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.562073 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.562215 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.565231 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:57.565237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:57.565240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:57.565309 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:57.569453 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.570431 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.570436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.570438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.570624 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.571162 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.571185 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.571188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.571546 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:57.571756 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.572638 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.572860 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 11:07:57.579021 19265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:07:57.586304 19265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:07:57.612737 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.601884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:44725 (local address 127.0.0.1:43190)
0925 11:07:57.602686 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.602730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.602936 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.603782 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.603790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.603822 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.604264 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.604280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.606676 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.606681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.609606 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.609618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.611620 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.611626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.611844 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.611900 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.612059 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.612283 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":5,"thread_start_us":1111,"threads_started":1}
I20260925 11:07:57.613448 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.602161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.8.181.1:44725)
0925 11:07:57.603073 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.603082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.603114 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:07:57.603238 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.603246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.603520 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.604518 (+ 998us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.604809 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.606385 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.609822 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.611387 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.611535 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.611792 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.612144 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.612328 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":594,"threads_started":1}
I20260925 11:07:57.614146 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.603813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:44599 (local address 127.0.0.1:50954)
0925 11:07:57.605556 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.605575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.605701 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.606667 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.606672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.606701 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.607226 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.607240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.608976 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.608981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.610852 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.610867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.613204 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.613210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.613372 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.613405 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.613509 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.613728 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1604,"mutex_wait_us":25,"thread_start_us":1475,"threads_started":1}
I20260925 11:07:57.614454 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.604317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.8.181.2:44599)
0925 11:07:57.605131 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.605138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.605845 (+ 707us) server_negotiation.cc:407] Connection header received
0925 11:07:57.605951 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.605960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.606202 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.606655 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.607418 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.608810 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.611043 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.612447 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.612577 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.613024 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.613948 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.614120 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":541,"threads_started":1}
I20260925 11:07:57.614506 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.603192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.8.181.3:35659)
0925 11:07:57.604874 (+ 1682us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.604885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.604921 (+ 36us) server_negotiation.cc:407] Connection header received
0925 11:07:57.605023 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.605031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.605328 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.606317 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.606630 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.608058 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.610692 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.611974 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.612115 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.613874 (+ 1759us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.613945 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.614118 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1495,"thread_start_us":1295,"threads_started":1}
I20260925 11:07:57.614982 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.602513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:35659 (local address 127.0.0.1:51146)
0925 11:07:57.603594 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.603612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.603722 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.605557 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.605564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.605599 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.606080 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.606093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.608217 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.608223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.610441 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.610456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.613048 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.613055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.613302 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.613349 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.613479 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.613619 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"spinlock_wait_cycles":15104,"thread_start_us":1236,"threads_started":1}
I20260925 11:07:57.618036 19523 tablet_service.cc:1467] Processing CreateTablet for tablet a377c608f0f14abca015c193e94d67a0 (DEFAULT_TABLE table=test-table [id=48a069cb9b0747c88c4b44c466b643ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:57.618023 19389 tablet_service.cc:1467] Processing CreateTablet for tablet a377c608f0f14abca015c193e94d67a0 (DEFAULT_TABLE table=test-table [id=48a069cb9b0747c88c4b44c466b643ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:57.618434 19657 tablet_service.cc:1467] Processing CreateTablet for tablet a377c608f0f14abca015c193e94d67a0 (DEFAULT_TABLE table=test-table [id=48a069cb9b0747c88c4b44c466b643ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:07:57.619625 19523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a377c608f0f14abca015c193e94d67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:57.619647 19389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a377c608f0f14abca015c193e94d67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:57.619800 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a377c608f0f14abca015c193e94d67a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:07:57.642771 19741 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Bootstrap starting.
I20260925 11:07:57.643289 19742 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Bootstrap starting.
I20260925 11:07:57.645905 19743 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Bootstrap starting.
I20260925 11:07:57.647706 19742 tablet_bootstrap.cc:654] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:57.649062 19741 tablet_bootstrap.cc:654] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:57.649253 19742 log.cc:824] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:57.649672 19743 tablet_bootstrap.cc:654] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Neither blocks nor log segments found. Creating new log.
I20260925 11:07:57.650585 19741 log.cc:824] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:57.650815 19743 log.cc:824] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Log is configured to *not* fsync() on all Append() calls
I20260925 11:07:57.655234 19741 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: No bootstrap required, opened a new log
I20260925 11:07:57.655607 19741 ts_tablet_manager.cc:1397] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260925 11:07:57.657692 19743 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: No bootstrap required, opened a new log
I20260925 11:07:57.657691 19742 tablet_bootstrap.cc:492] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: No bootstrap required, opened a new log
I20260925 11:07:57.657975 19742 ts_tablet_manager.cc:1397] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260925 11:07:57.658105 19743 ts_tablet_manager.cc:1397] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
I20260925 11:07:57.665875 19742 raft_consensus.cc:348] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.666322 19742 raft_consensus.cc:374] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:57.666441 19742 raft_consensus.cc:729] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205bd55b62e74f16a405f36a5e2a9f12, State: Initialized, Role: FOLLOWER
I20260925 11:07:57.667198 19742 consensus_queue.cc:260] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.667107 19741 raft_consensus.cc:348] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.667644 19741 raft_consensus.cc:374] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:57.667784 19741 raft_consensus.cc:729] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b2e75c2dc04f168f12e62ecbd51072, State: Initialized, Role: FOLLOWER
I20260925 11:07:57.668574 19741 consensus_queue.cc:260] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.669134 19743 raft_consensus.cc:348] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.669915 19743 raft_consensus.cc:374] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:07:57.670128 19743 raft_consensus.cc:729] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f3af3026c34226ba98ca9c73a99d03, State: Initialized, Role: FOLLOWER
I20260925 11:07:57.671262 19742 ts_tablet_manager.cc:1428] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260925 11:07:57.671149 19743 consensus_queue.cc:260] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.672442 19588 heartbeater.cc:503] Master 127.8.181.62:32889 was elected leader, sending a full tablet report...
I20260925 11:07:57.673411 19741 ts_tablet_manager.cc:1428] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260925 11:07:57.674831 19722 heartbeater.cc:503] Master 127.8.181.62:32889 was elected leader, sending a full tablet report...
I20260925 11:07:57.677222 19743 ts_tablet_manager.cc:1428] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260925 11:07:57.692836 19748 raft_consensus.cc:482] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:57.693171 19748 raft_consensus.cc:504] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.695111 19748 leader_election.cc:288] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 205bd55b62e74f16a405f36a5e2a9f12 (127.8.181.1:44725), a4f3af3026c34226ba98ca9c73a99d03 (127.8.181.3:35659)
I20260925 11:07:57.697608 19747 raft_consensus.cc:482] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:07:57.697947 19747 raft_consensus.cc:504] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.700332 19747 leader_election.cc:288] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b2e75c2dc04f168f12e62ecbd51072 (127.8.181.2:44599), a4f3af3026c34226ba98ca9c73a99d03 (127.8.181.3:35659)
I20260925 11:07:57.706336 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.695577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:43055 (local address 127.8.181.1:44725)
0925 11:07:57.695838 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.695848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.696618 (+ 770us) server_negotiation.cc:407] Connection header received
0925 11:07:57.696881 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.696895 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.696959 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.697102 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.697861 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.699309 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.702522 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.704547 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.704715 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.704881 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.705707 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.705918 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 11:07:57.707430 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.695539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:44725 (local address 127.8.181.2:43055)
0925 11:07:57.696468 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.696487 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.696709 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.697116 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.697121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.697146 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.697649 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.697670 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.699501 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.699506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.702243 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.702268 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.705150 (+ 2882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.705160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.705434 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.705493 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.706680 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.706913 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":519,"threads_started":1}
I20260925 11:07:57.707762 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.695689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35327 (local address 127.8.181.3:35659)
0925 11:07:57.695932 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.695942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.696712 (+ 770us) server_negotiation.cc:407] Connection header received
0925 11:07:57.696919 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.696924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.696988 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.697111 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.697988 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.700282 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.703955 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.705899 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.706070 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.706261 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.707194 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.707409 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:07:57.708921 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.695665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:35659 (local address 127.8.181.2:35327)
0925 11:07:57.696600 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.696617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.696742 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.697147 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.697152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.697181 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.697772 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.697785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.700479 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.700489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.703641 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.703668 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.706595 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.706606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.706904 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.706957 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.708170 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.708381 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":4,"thread_start_us":543,"threads_started":1}
I20260925 11:07:57.708910 19409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205bd55b62e74f16a405f36a5e2a9f12" is_pre_election: true
I20260925 11:07:57.709604 19409 raft_consensus.cc:2454] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b2e75c2dc04f168f12e62ecbd51072 in term 0.
I20260925 11:07:57.710073 19677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f3af3026c34226ba98ca9c73a99d03" is_pre_election: true
I20260925 11:07:57.710738 19677 raft_consensus.cc:2454] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b2e75c2dc04f168f12e62ecbd51072 in term 0.
I20260925 11:07:57.711477 19478 leader_election.cc:302] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 205bd55b62e74f16a405f36a5e2a9f12, a0b2e75c2dc04f168f12e62ecbd51072, a4f3af3026c34226ba98ca9c73a99d03; no voters:
I20260925 11:07:57.711983 19748 raft_consensus.cc:2781] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:07:57.712136 19748 raft_consensus.cc:482] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:07:57.712245 19748 raft_consensus.cc:3037] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:57.715988 19748 raft_consensus.cc:504] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.717363 19677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f3af3026c34226ba98ca9c73a99d03"
I20260925 11:07:57.717383 19409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205bd55b62e74f16a405f36a5e2a9f12"
I20260925 11:07:57.717623 19409 raft_consensus.cc:3037] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:57.717614 19677 raft_consensus.cc:3037] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:07:57.718290 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.701535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:55985 (local address 127.8.181.3:35659)
0925 11:07:57.708027 (+ 6492us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.708033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.710790 (+ 2757us) server_negotiation.cc:407] Connection header received
0925 11:07:57.711124 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.711131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.711237 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.711364 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.712166 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.713344 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.715530 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.716818 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.716966 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.717132 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.717739 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.717900 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6378,"thread_start_us":6379,"threads_started":1}
I20260925 11:07:57.718683 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.701464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:37115 (local address 127.8.181.2:44599)
0925 11:07:57.701669 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.701675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.710349 (+ 8674us) server_negotiation.cc:407] Connection header received
0925 11:07:57.710435 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.710443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.710589 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.710692 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:07:57.711503 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.712863 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.715903 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.717374 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.717513 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.717626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.718193 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.718365 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:07:57.719241 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.701053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:35659 (local address 127.8.181.1:55985)
0925 11:07:57.710693 (+ 9640us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.710712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.710893 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.711531 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.711536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.711564 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.711949 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.711961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.713496 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.713502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.715338 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.715353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.717305 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.717312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.717526 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.717563 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.718584 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.718767 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9515,"mutex_wait_us":3,"thread_start_us":9295,"threads_started":1}
I20260925 11:07:57.719482 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.700832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:44599 (local address 127.8.181.1:37115)
0925 11:07:57.709931 (+ 9099us) negotiation.cc:107] Waiting for socket to connect
0925 11:07:57.709953 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:07:57.710103 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:07:57.710868 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:07:57.710874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:07:57.710905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:07:57.711323 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.711341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.713025 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.713031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.715705 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:07:57.715724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.717798 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.717806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:07:57.717991 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.718027 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 11:07:57.718840 (+ 813us) client_negotiation.cc:240] Negotiation successful
0925 11:07:57.719002 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8947,"mutex_wait_us":165,"thread_start_us":9879,"threads_started":1}
I20260925 11:07:57.719871 19748 leader_election.cc:288] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [CANDIDATE]: Term 1 election: Requested vote from peers 205bd55b62e74f16a405f36a5e2a9f12 (127.8.181.1:44725), a4f3af3026c34226ba98ca9c73a99d03 (127.8.181.3:35659)
I20260925 11:07:57.721124 19543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "205bd55b62e74f16a405f36a5e2a9f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" is_pre_election: true
I20260925 11:07:57.721447 19676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a377c608f0f14abca015c193e94d67a0" candidate_uuid: "205bd55b62e74f16a405f36a5e2a9f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4f3af3026c34226ba98ca9c73a99d03" is_pre_election: true
I20260925 11:07:57.721940 19543 raft_consensus.cc:2379] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 205bd55b62e74f16a405f36a5e2a9f12 in current term 1: Already voted for candidate a0b2e75c2dc04f168f12e62ecbd51072 in this term.
I20260925 11:07:57.722920 19677 raft_consensus.cc:2454] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b2e75c2dc04f168f12e62ecbd51072 in term 1.
I20260925 11:07:57.723215 19409 raft_consensus.cc:2454] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b2e75c2dc04f168f12e62ecbd51072 in term 1.
I20260925 11:07:57.723223 19345 leader_election.cc:302] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 205bd55b62e74f16a405f36a5e2a9f12; no voters: a0b2e75c2dc04f168f12e62ecbd51072, a4f3af3026c34226ba98ca9c73a99d03
I20260925 11:07:57.723438 19479 leader_election.cc:302] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b2e75c2dc04f168f12e62ecbd51072, a4f3af3026c34226ba98ca9c73a99d03; no voters:
I20260925 11:07:57.723718 19747 raft_consensus.cc:2726] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 11:07:57.723896 19748 raft_consensus.cc:2781] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:07:57.724453 19748 raft_consensus.cc:686] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 LEADER]: Becoming Leader. State: Replica: a0b2e75c2dc04f168f12e62ecbd51072, State: Running, Role: LEADER
I20260925 11:07:57.724963 19748 consensus_queue.cc:237] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } }
I20260925 11:07:57.730276 19265 catalog_manager.cc:5495] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0b2e75c2dc04f168f12e62ecbd51072 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0b2e75c2dc04f168f12e62ecbd51072" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 } health_report { overall_health: UNKNOWN } } }
W20260925 11:07:57.760147 19723 tablet.cc:2369] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:07:57.762229 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.755535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.8.181.2:44599)
0925 11:07:57.755757 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.755763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.755920 (+ 157us) server_negotiation.cc:407] Connection header received
0925 11:07:57.756172 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.756176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.756274 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.756438 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:07:57.757002 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.757788 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.759327 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.760028 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.760171 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.760950 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:07:57.761502 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:07:57.761635 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.761870 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.761951 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:07:57.772233 19677 raft_consensus.cc:1261] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 1 FOLLOWER]: Refusing update from remote peer a0b2e75c2dc04f168f12e62ecbd51072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:57.772291 19409 raft_consensus.cc:1261] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 1 FOLLOWER]: Refusing update from remote peer a0b2e75c2dc04f168f12e62ecbd51072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:07:57.773138 19755 consensus_queue.cc:1035] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4f3af3026c34226ba98ca9c73a99d03" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 35659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:57.773649 19748 consensus_queue.cc:1035] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "205bd55b62e74f16a405f36a5e2a9f12" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:07:57.786715 19758 mvcc.cc:205] Tried to move back new op lower bound from 7333210020942995456 to 7333210020765339648. Current Snapshot: MvccSnapshot[applied={T|T < 7333210020942995456}]
I20260925 11:07:57.804490 19265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:57.804801 19265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:07:57.808818 19265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 823ad5a6436143799a053577439bad41: Sending DeleteTablet for 3 replicas of tablet a377c608f0f14abca015c193e94d67a0
I20260925 11:07:57.810220 19657 tablet_service.cc:1514] Processing DeleteTablet for tablet a377c608f0f14abca015c193e94d67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:57 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:51146
I20260925 11:07:57.810321 19523 tablet_service.cc:1514] Processing DeleteTablet for tablet a377c608f0f14abca015c193e94d67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:57 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260925 11:07:57.810580 19389 tablet_service.cc:1514] Processing DeleteTablet for tablet a377c608f0f14abca015c193e94d67a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:07:57 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:43190
I20260925 11:07:57.812747 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 19326
I20260925 11:07:57.822738 19768 tablet_replica.cc:330] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: stopping tablet replica
I20260925 11:07:57.823417 19768 raft_consensus.cc:2229] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:57.824090 19768 raft_consensus.cc:2258] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:57.826648 19768 ts_tablet_manager.cc:1905] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:57.834566 19768 ts_tablet_manager.cc:1918] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:57.834869 19768 log.cc:1196] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/a377c608f0f14abca015c193e94d67a0
I20260925 11:07:57.835517 19768 ts_tablet_manager.cc:1939] T a377c608f0f14abca015c193e94d67a0 P a4f3af3026c34226ba98ca9c73a99d03: Deleting consensus metadata
I20260925 11:07:57.837819 19252 catalog_manager.cc:4841] TS a4f3af3026c34226ba98ca9c73a99d03 (127.8.181.3:35659): tablet a377c608f0f14abca015c193e94d67a0 (table test-table [id=48a069cb9b0747c88c4b44c466b643ef]) successfully deleted
I20260925 11:07:57.839020 19769 tablet_replica.cc:330] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: stopping tablet replica
I20260925 11:07:57.839597 19769 raft_consensus.cc:2229] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:07:57.840412 19769 raft_consensus.cc:2258] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:57.843219 19769 ts_tablet_manager.cc:1905] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:57.849788 19770 tablet_replica.cc:330] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: stopping tablet replica
I20260925 11:07:57.850309 19770 raft_consensus.cc:2229] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:07:57.851034 19770 raft_consensus.cc:2258] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:07:57.851426 19769 ts_tablet_manager.cc:1918] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:57.851603 19769 log.cc:1196] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/a377c608f0f14abca015c193e94d67a0
I20260925 11:07:57.852174 19769 ts_tablet_manager.cc:1939] T a377c608f0f14abca015c193e94d67a0 P a0b2e75c2dc04f168f12e62ecbd51072: Deleting consensus metadata
I20260925 11:07:57.853574 19770 ts_tablet_manager.cc:1905] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:07:57.854064 19251 catalog_manager.cc:4841] TS a0b2e75c2dc04f168f12e62ecbd51072 (127.8.181.2:44599): tablet a377c608f0f14abca015c193e94d67a0 (table test-table [id=48a069cb9b0747c88c4b44c466b643ef]) successfully deleted
Sep 25 11:07:57 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:07:57.860807 19770 ts_tablet_manager.cc:1918] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:07:57.860989 19770 log.cc:1196] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/a377c608f0f14abca015c193e94d67a0
I20260925 11:07:57.861567 19770 ts_tablet_manager.cc:1939] T a377c608f0f14abca015c193e94d67a0 P 205bd55b62e74f16a405f36a5e2a9f12: Deleting consensus metadata
I20260925 11:07:57.863442 19251 catalog_manager.cc:4841] TS 205bd55b62e74f16a405f36a5e2a9f12 (127.8.181.1:44725): tablet a377c608f0f14abca015c193e94d67a0 (table test-table [id=48a069cb9b0747c88c4b44c466b643ef]) successfully deleted
I20260925 11:07:57.864761 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:57.815045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.8.181.1:44725)
0925 11:07:57.815276 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:57.815286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:57.839089 (+ 23803us) server_negotiation.cc:407] Connection header received
0925 11:07:57.839579 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:57.839588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:57.839721 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:57.839820 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:57.842496 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.846676 (+ 4180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.850097 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:57.850931 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:57.851092 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:57.856592 (+ 5500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:57.856605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:57.856624 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:57.856672 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:57.860857 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.861912 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.861923 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.861927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.862098 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:57.862595 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:57.862600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:57.862604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:57.863098 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:57.863423 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:57.864143 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 11:07:57.864344 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:07:57.865545 19449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:57.989066 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 19326
I20260925 11:07:58.007980 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 19460
Sep 25 11:07:58 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:07:58.028106 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:58.008783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.8.181.2:44599)
0925 11:07:58.008985 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:58.008992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:58.009013 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:07:58.009170 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:58.009177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:58.009284 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:58.009395 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:58.010864 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:58.012017 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:58.013383 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:58.014084 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:58.014231 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:58.019240 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:58.019260 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:58.019283 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:58.019353 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:58.024377 (+ 5024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:58.025411 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:58.025421 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:58.025425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:58.025667 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:58.026296 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:58.026301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:58.026305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:58.026870 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:58.027168 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:58.027491 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 11:07:58.027747 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 11:07:58.028966 19583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:58.146380 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 19460
I20260925 11:07:58.167012 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 19594
Sep 25 11:07:58 dist-test-slave-6042 krb5kdc[19220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:07:58.184337 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:07:58.167866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.8.181.3:35659)
0925 11:07:58.168061 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 11:07:58.168067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:07:58.168087 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:07:58.168261 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:07:58.168267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:07:58.168350 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:07:58.168475 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:07:58.169767 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:58.170543 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:58.171830 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:07:58.172277 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:07:58.172432 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:07:58.177558 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:07:58.177576 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:07:58.177594 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:07:58.177665 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:07:58.180831 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:58.181764 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:58.181773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:58.181776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:58.181910 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:07:58.182371 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:07:58.182376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:07:58.182379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:07:58.182965 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:07:58.183340 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0925 11:07:58.183780 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 11:07:58.184033 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 11:07:58.185055 19717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:58.279954 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 19594
I20260925 11:07:58.297078 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 19233
I20260925 11:07:58.298218 19295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:07:58.411001 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 19233
2026-09-25T11:07:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4730 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:07:58 dist-test-slave-6042 krb5kdc[19778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:07:58 dist-test-slave-6042 krb5kdc[19778](info): set up 2 sockets
Sep 25 11:07:58 dist-test-slave-6042 krb5kdc[19778](info): commencing operation
krb5kdc: starting...
W20260925 11:07:59.588181 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:07:59 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:07:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangeradmin_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangeradmin_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangerlookup_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangerlookup_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
I20260925 11:07:59.967309 8916 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskA5RC6A/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/postgres -l logfile start
2026-09-25 11:08:03.121 UTC [19808] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 11:08:03.121 UTC [19808] LOG: listening on IPv4 address "127.8.181.20", port 42217
2026-09-25 11:08:03.126 UTC [19808] LOG: listening on Unix socket "/tmp/.s.PGSQL.42217"
2026-09-25 11:08:03.152 UTC [19811] LOG: database system was shut down at 2026-09-25 11:08:01 UTC
2026-09-25 11:08:03.156 UTC [19808] LOG: database system is ready to accept connections
I20260925 11:08:04.175694 8916 mini_postgres.cc:96] Postgres bound to 42217
2026-09-25 11:08:04.194 UTC [19819] FATAL: database "slave" does not exist
127.8.181.20:42217 - accepting connections
I20260925 11:08:04.195225 8916 mini_ranger.cc:161] Starting Ranger...
I20260925 11:08:04.224399 8916 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 11:08:04.892019 8916 mini_ranger.cc:87] Created ranger Postgres database
I20260925 11:08:04.892218 8916 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin
2026-09-25 11:08:05,198 [I] DB FLAVOR :POSTGRES
2026-09-25 11:08:05,199 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 11:08:05,199 [I] Checking connection..
2026-09-25 11:08:05,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 11:08:05,469 [I] Checking connection passed.
2026-09-25 11:08:05,470 [I] --------- Verifying version history table ---------
2026-09-25 11:08:05,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:08:05,729 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 11:08:05,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:08:05,999 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 11:08:05,999 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 11:08:05,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 11:08:06.253 UTC [19883] WARNING: there is no transaction in progress
2026-09-25 11:08:06,260 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 11:08:06,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:08:06,515 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 11:08:06,515 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 11:08:06,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 11:08:06,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 11:08:07,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6042.c.gcp-upstream.internal','N') ;"
2026-09-25 11:08:07,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 11:08:07,555 [I] Table x_portal_user does not exist in database ranger
2026-09-25 11:08:07,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 11:08:07,818 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 11:08:07,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 11:08:08,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 11:08:08,329 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 11:08:08,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 11:08:09.492 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.511 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.534 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.554 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.578 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.599 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.664 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.673 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.691 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.703 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.715 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.732 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.744 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.752 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.762 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:09.771 UTC [19991] WARNING: there is no transaction in progress
2026-09-25 11:08:10,646 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 11:08:10,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 11:08:10,908 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 11:08:10,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 11:08:11,194 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 11:08:11,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 11:08:11,437 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 11:08:11,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 11:08:11,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 11:08:11,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6042.c.gcp-upstream.internal';"
2026-09-25 11:08:12,177 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 11:08:12,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 11:08:12,447 [I] DB_PATCHES have already been applied
I20260925 11:08:12.453068 8916 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskA5RC6A/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 11:08:12.453187 8916 mini_ranger.cc:193] Using host: 127.8.181.20
I20260925 11:08:12.469133 8916 mini_ranger.cc:227] Ranger admin URL: http://127.8.181.20:37341
Sep 25, 2026 11:08:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 11:08:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 11:08:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 11:08:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 11:08:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 11:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 11:08:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 11:08:13.673040 8916 mini_ranger.cc:160] Time spent starting Ranger: real 9.478s user 0.000s sys 0.048s
Sep 25, 2026 11:08:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 11:08:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 11:08:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 11:08:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 11:08:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 11:08:37.272372 8916 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangerkms_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/rangerkms_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.181.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.181.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 11:08:37 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334517, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 11:08:37.450598 8916 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 11:08:37.474675 8916 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 11:08:38.618065 8916 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 11:08:38.618332 8916 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms
I20260925 11:08:38.618539 8916 mini_ranger_kms.cc:227] Using postgres at 127.8.181.20:42217
2026-09-25 11:08:39,075 [I] DB FLAVOR :POSTGRES
2026-09-25 11:08:39,075 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 11:08:39,075 [I] Checking connection
2026-09-25 11:08:39,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 11:08:39,339 [I] connection success
2026-09-25 11:08:39,339 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 11:08:39,339 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 11:08:39,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 11:08:39,614 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 11:08:39,615 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 11:08:39,615 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 11:08:39,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:42217/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 11:08:39,882 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 11:08:40.369591 8916 mini_ranger_kms.cc:310] Created kms service
I20260925 11:08:40.598217 8916 mini_ranger_kms.cc:326] Created kudu user
I20260925 11:08:40.648452 8916 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 11:08:40 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334517, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:41.072227 8916 mini_ranger_kms.cc:384] Added ranger policy
I20260925 11:08:41.072562 8916 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms:/tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 11:08:41.072628 8916 mini_ranger_kms.cc:240] Using host: 127.8.181.20
I20260925 11:08:41.095656 8916 mini_ranger_kms.cc:276] Ranger KMS PID: 20172
I20260925 11:08:41.095861 8916 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.181.20:50903
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 11:08:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 11:08:43 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790334523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:08:43.419862 8916 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.969s user 0.131s sys 0.332s
I20260925 11:08:43.420120 8916 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 11:08:43.420331 8916 mini_ranger_kms.cc:401] 127.8.181.20:50903/kms/v1/keys
Sep 25 11:08:43 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
Sep 25 11:08:44 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334523, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
Sep 25 11:08:46 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:08:46.762373 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41339
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:41339
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:50903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:08:47.110150 20222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:08:47.110585 20222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:08:47.110688 20222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:08:47.118330 20222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:08:47.118458 20222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:08:47.118530 20222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:08:47.118582 20222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:08:47.118631 20222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:08:47.129423 20222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.181.20:50903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:41339
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:08:47.131085 20222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:08:47.132521 20222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:08:47.146041 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:47.146064 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:47.145991 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:08:47.146445 20222 server_base.cc:1034] running on GCE node
I20260925 11:08:47.147485 20222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:08:47.148911 20222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:08:47.150152 20222 hybrid_clock.cc:648] HybridClock initialized: now 1790334527150105 us; error 35 us; skew 500 ppm
I20260925 11:08:47.150595 20222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:08:47 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:08:47.154042 20222 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:08:47.155575 20222 webserver.cc:460] Webserver started at http://127.8.181.62:44347/ using document root <none> and password file <none>
I20260925 11:08:47.156286 20222 fs_manager.cc:359] Metadata directory not provided
I20260925 11:08:47.156383 20222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:08:47.156718 20222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:08:47 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:47.213768 20222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "637f936006254b9dad2a4510fd513962"
format_stamp: "Formatted at 2026-09-25 11:08:47 on dist-test-slave-6042"
server_key: "e463951d966cde33ee4970fc8848c56c"
server_key_iv: "14447cf4d821a3f99991293865164f5e"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:47.214768 20222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "637f936006254b9dad2a4510fd513962"
format_stamp: "Formatted at 2026-09-25 11:08:47 on dist-test-slave-6042"
server_key: "e463951d966cde33ee4970fc8848c56c"
server_key_iv: "14447cf4d821a3f99991293865164f5e"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:47.221140 20222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:08:47.243132 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:47.244870 20222 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.004s
I20260925 11:08:47.245100 20222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "637f936006254b9dad2a4510fd513962"
format_stamp: "Formatted at 2026-09-25 11:08:47 on dist-test-slave-6042"
server_key: "e463951d966cde33ee4970fc8848c56c"
server_key_iv: "14447cf4d821a3f99991293865164f5e"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:47.245329 20222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:08:47.261469 20222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:08:47.266016 20222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:08:47.266383 20222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:08:47.287096 20222 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:41339
I20260925 11:08:47.287128 20288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:41339 every 8 connection(s)
I20260925 11:08:47.288511 20222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:08:47.291016 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 20222
I20260925 11:08:47.291241 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:08:47.292079 8916 external_mini_cluster.cc:1428] Setting key ce49bf37bc46f419c4635ad6a262ef46
I20260925 11:08:47.293437 20289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 11:08:47 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:08:47.306799 20289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: Bootstrap starting.
I20260925 11:08:47.310736 20289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: Neither blocks nor log segments found. Creating new log.
I20260925 11:08:47.311959 20289 log.cc:824] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: Log is configured to *not* fsync() on all Append() calls
I20260925 11:08:47.315112 20289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: No bootstrap required, opened a new log
I20260925 11:08:47.315778 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:47.294288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.8.181.62:41339)
0925 11:08:47.295166 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:47.295179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:47.295221 (+ 42us) server_negotiation.cc:407] Connection header received
0925 11:08:47.296973 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:47.296997 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:47.297437 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:47.297908 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:47.299405 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:47.300817 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:47.302094 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:47.302929 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:47.303139 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:47.307589 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:47.307598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:47.307614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:47.307656 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:47.311190 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:47.312049 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:47.312056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:47.312064 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:47.312281 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:47.312705 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:47.312710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:47.312714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:47.313309 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:47.313633 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:47.314111 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 11:08:47.314594 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":447,"threads_started":1}
I20260925 11:08:47.323385 20289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } }
I20260925 11:08:47.323863 20289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:08:47.323978 20289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637f936006254b9dad2a4510fd513962, State: Initialized, Role: FOLLOWER
I20260925 11:08:47.324575 20289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } }
I20260925 11:08:47.324836 20289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:08:47.324990 20289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:08:47.325181 20289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:08:47.327733 20289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } }
I20260925 11:08:47.328341 20289 leader_election.cc:302] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 637f936006254b9dad2a4510fd513962; no voters:
I20260925 11:08:47.329042 20289 leader_election.cc:288] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:08:47.329201 20294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:08:47.330152 20294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [term 1 LEADER]: Becoming Leader. State: Replica: 637f936006254b9dad2a4510fd513962, State: Running, Role: LEADER
I20260925 11:08:47.330806 20294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } }
I20260925 11:08:47.331517 20289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:08:47.334105 20296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637f936006254b9dad2a4510fd513962. Latest consensus state: current_term: 1 leader_uuid: "637f936006254b9dad2a4510fd513962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } } }
I20260925 11:08:47.334600 20296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [sys.catalog]: This master's current role is: LEADER
I20260925 11:08:47.334075 20295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "637f936006254b9dad2a4510fd513962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "637f936006254b9dad2a4510fd513962" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41339 } } }
I20260925 11:08:47.335068 20295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962 [sys.catalog]: This master's current role is: LEADER
I20260925 11:08:47.336553 20300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:08:47.341379 20300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:08:47.350661 20300 catalog_manager.cc:1347] Generated new cluster ID: 3fde96689fa34c828126fab7a3b81f12
I20260925 11:08:47.350803 20300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:08:47.362483 20289 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 11:08:47.371667 20300 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:08:47.373417 20300 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:08:47.384254 20300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: Generated new TSK 0
I20260925 11:08:47.385223 20300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:08:48.306164 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:50903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:08:48 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 11:08:48.736989 20334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:08:48.737450 20334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:08:48.737617 20334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:08:48.749302 20334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:08:48.749500 20334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:08:48.749730 20334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:08:48.768864 20334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.181.20:50903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:08:48.770993 20334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:08:48.773077 20334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:08:48.800004 20344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:48.800865 20345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:48.820724 20347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:08:48.822116 20334 server_base.cc:1034] running on GCE node
I20260925 11:08:48.823472 20334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:08:48.825356 20334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:08:48.826666 20334 hybrid_clock.cc:648] HybridClock initialized: now 1790334528826526 us; error 71 us; skew 500 ppm
I20260925 11:08:48.827330 20334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:08:48 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:08:48.840232 20334 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:08:48.843551 20334 webserver.cc:460] Webserver started at http://127.8.181.1:39693/ using document root <none> and password file <none>
I20260925 11:08:48.844599 20334 fs_manager.cc:359] Metadata directory not provided
I20260925 11:08:48.844781 20334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:08:48.845146 20334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:08:48 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:48.868323 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d"
format_stamp: "Formatted at 2026-09-25 11:08:48 on dist-test-slave-6042"
server_key: "d28c8684394188771c00ddb54cf21409"
server_key_iv: "c2baad6bf901283d6c3ad04a1f067825"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:48.869779 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d"
format_stamp: "Formatted at 2026-09-25 11:08:48 on dist-test-slave-6042"
server_key: "d28c8684394188771c00ddb54cf21409"
server_key_iv: "c2baad6bf901283d6c3ad04a1f067825"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:48.883363 20334 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.015s sys 0.001s
I20260925 11:08:48.919309 20354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:48.921223 20334 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.011s sys 0.008s
I20260925 11:08:48.921452 20334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d"
format_stamp: "Formatted at 2026-09-25 11:08:48 on dist-test-slave-6042"
server_key: "d28c8684394188771c00ddb54cf21409"
server_key_iv: "c2baad6bf901283d6c3ad04a1f067825"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:48.921731 20334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:08:48.959977 20334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:08:48.965783 20334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:08:48.966204 20334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:08:48.967514 20334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:08:48.969357 20334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:08:48.969485 20334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:48.969640 20334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:08:48.969736 20334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:08:49 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:49.297009 20334 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:41249
I20260925 11:08:49.297078 20466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:41249 every 8 connection(s)
I20260925 11:08:49.298929 20334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:08:49.309293 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 20334
I20260925 11:08:49.309511 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:08:49.310587 8916 external_mini_cluster.cc:1428] Setting key f8a6acae136ba25d362af79f66d83e23
Sep 25 11:08:49 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:08:49.346050 20469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:49.303062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:40515 (local address 127.8.181.62:41339)
0925 11:08:49.323198 (+ 20136us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:49.323203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:49.323224 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:08:49.323289 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:49.323294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:49.323378 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:49.323547 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:49.326291 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:49.327851 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:49.331111 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:49.331940 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:49.332061 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:49.337507 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:49.337513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:49.337517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:49.337553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:49.341724 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:49.342850 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:49.342855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:49.342857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:49.343048 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:49.343762 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:49.343771 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:49.343773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:49.344127 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:49.344291 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:49.345195 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 11:08:49.345431 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19981,"thread_start_us":19788,"threads_started":1}
I20260925 11:08:49.347347 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:49.303806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41339 (local address 127.8.181.1:40515)
0925 11:08:49.318375 (+ 14569us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:49.318454 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:49.322067 (+ 3613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:49.323938 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:49.323953 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:49.324825 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:08:49.326039 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:49.326068 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:49.328104 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:49.328110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:49.330888 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:49.330934 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:49.332562 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:49.332570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:49.332920 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:49.333859 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:08:49.333883 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:08:49.337091 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:08:49.341923 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:49.341935 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:49.341955 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:49.342691 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:49.343208 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:49.343212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:49.343215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:49.343606 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:49.344319 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:08:49.344328 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:08:49.344792 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 11:08:49.345107 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 11:08:49.345827 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14072,"thread_start_us":18808,"threads_started":1}
I20260925 11:08:49.349588 20467 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41339
I20260925 11:08:49.350039 20467 heartbeater.cc:464] Registering TS with master...
I20260925 11:08:49.351219 20467 heartbeater.cc:511] Master 127.8.181.62:41339 requested a full tablet report, sending...
I20260925 11:08:49.353636 20254 ts_manager.cc:194] Registered new tserver with Master: 6e2e15bc1ba84cdab1ef67b2bf4bf50d (127.8.181.1:41249)
I20260925 11:08:49.355535 20254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:40515
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:08:49.403090 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:50903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:08:49 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334528, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
W20260925 11:08:49.905819 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:08:49.906126 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:08:49.906219 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:08:49.914258 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:08:49.914397 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:08:49.914618 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:08:49.924928 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.181.20:50903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:08:49.926265 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:08:49.927711 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:08:49.942065 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:49.942109 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:49.942137 20490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:08:49.943874 20473 server_base.cc:1034] running on GCE node
I20260925 11:08:49.944504 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:08:49.945484 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:08:49.946686 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790334529946646 us; error 34 us; skew 500 ppm
I20260925 11:08:49.947106 20473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:08:49 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:08:49.950950 20473 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:08:49.952698 20473 webserver.cc:460] Webserver started at http://127.8.181.2:43475/ using document root <none> and password file <none>
I20260925 11:08:49.953429 20473 fs_manager.cc:359] Metadata directory not provided
I20260925 11:08:49.953595 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:08:49.953936 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:08:49 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:49.971467 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "74573d0c40514c378564f08d44699c91"
format_stamp: "Formatted at 2026-09-25 11:08:49 on dist-test-slave-6042"
server_key: "596b773997a24923937d431ed1e5e87f"
server_key_iv: "d0d2c558ffedb173e17ba8c52f27fcea"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:49.972493 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "74573d0c40514c378564f08d44699c91"
format_stamp: "Formatted at 2026-09-25 11:08:49 on dist-test-slave-6042"
server_key: "596b773997a24923937d431ed1e5e87f"
server_key_iv: "d0d2c558ffedb173e17ba8c52f27fcea"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:49.978559 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 11:08:49.996781 20497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:49.998943 20473 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:08:49.999168 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "74573d0c40514c378564f08d44699c91"
format_stamp: "Formatted at 2026-09-25 11:08:49 on dist-test-slave-6042"
server_key: "596b773997a24923937d431ed1e5e87f"
server_key_iv: "d0d2c558ffedb173e17ba8c52f27fcea"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:49.999465 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:08:50.027464 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:08:50.033867 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:08:50.034256 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:08:50.035588 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:08:50.037446 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:08:50.037546 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:50.037671 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:08:50.037746 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:50.081960 20473 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:34473
I20260925 11:08:50.081974 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:34473 every 8 connection(s)
I20260925 11:08:50.083709 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:08:50.084854 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 20473
I20260925 11:08:50.085031 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:08:50.085646 8916 external_mini_cluster.cc:1428] Setting key 73415d13bd886309b9576934fbcfc255
Sep 25 11:08:50 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334529, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:08:50.118232 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.089056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:34377 (local address 127.8.181.62:41339)
0925 11:08:50.098050 (+ 8994us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.098060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.098088 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:08:50.098171 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.098175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.098243 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.098403 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:50.100462 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.101694 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.104676 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.105356 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.105504 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.110704 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:50.110715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:50.110723 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:50.110783 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:50.114757 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.115586 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.115590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.115593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.115730 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.116319 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.116323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.116325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.116724 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:50.116842 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.117634 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.117958 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8819,"thread_start_us":8715,"threads_started":1}
I20260925 11:08:50.119686 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.087606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41339 (local address 127.8.181.2:34377)
0925 11:08:50.088570 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.088623 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.091659 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.098781 (+ 7122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.098793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.099282 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:08:50.100196 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.100228 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.102022 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.102031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.104478 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.104501 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.105892 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.105900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.106230 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.107115 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:08:50.107145 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:08:50.110260 (+ 3115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:08:50.114943 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:50.114953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:50.114968 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:50.115450 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:50.115836 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:50.115839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:50.115841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:50.116170 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:50.116860 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:08:50.116866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:08:50.117237 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.117472 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.118121 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":2489,"threads_started":1}
I20260925 11:08:50.121344 20610 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41339
I20260925 11:08:50.121702 20610 heartbeater.cc:464] Registering TS with master...
I20260925 11:08:50.122529 20610 heartbeater.cc:511] Master 127.8.181.62:41339 requested a full tablet report, sending...
I20260925 11:08:50.124560 20254 ts_manager.cc:194] Registered new tserver with Master: 74573d0c40514c378564f08d44699c91 (127.8.181.2:34473)
I20260925 11:08:50.125819 20254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:34377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:08:50.151822 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:50903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:08:50.359496 20467 heartbeater.cc:503] Master 127.8.181.62:41339 was elected leader, sending a full tablet report...
W20260925 11:08:50.517026 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:08:50.517388 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:08:50.517566 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:08:50.527410 20616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:08:50.527585 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:08:50.527839 20616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:08:50.541038 20616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.181.20:50903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:08:50.542317 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:08:50.543773 20616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:08:50.557732 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:50.557845 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:08:50.557943 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:08:50.559316 20616 server_base.cc:1034] running on GCE node
I20260925 11:08:50.559998 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:08:50.561021 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:08:50.562245 20616 hybrid_clock.cc:648] HybridClock initialized: now 1790334530562173 us; error 68 us; skew 500 ppm
I20260925 11:08:50.562618 20616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:08:50 dist-test-slave-6042 krb5kdc[19778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334530, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:08:50.565977 20616 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:08:50.567710 20616 webserver.cc:460] Webserver started at http://127.8.181.3:43029/ using document root <none> and password file <none>
I20260925 11:08:50.568446 20616 fs_manager.cc:359] Metadata directory not provided
I20260925 11:08:50.568598 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:08:50.568918 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:08:50 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334530, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:08:50.584533 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d"
format_stamp: "Formatted at 2026-09-25 11:08:50 on dist-test-slave-6042"
server_key: "1ae302f12e3c6838597d67fd0580b213"
server_key_iv: "ccca136b93a7dbfe0e8d7dc1d14e4a35"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:50.585520 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d"
format_stamp: "Formatted at 2026-09-25 11:08:50 on dist-test-slave-6042"
server_key: "1ae302f12e3c6838597d67fd0580b213"
server_key_iv: "ccca136b93a7dbfe0e8d7dc1d14e4a35"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:50.591389 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 11:08:50.609720 20631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:50.611379 20616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:08:50.611588 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d"
format_stamp: "Formatted at 2026-09-25 11:08:50 on dist-test-slave-6042"
server_key: "1ae302f12e3c6838597d67fd0580b213"
server_key_iv: "ccca136b93a7dbfe0e8d7dc1d14e4a35"
server_key_version: "kuduclusterkey@0"
I20260925 11:08:50.611822 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:08:50.632349 20616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:08:50.636896 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:08:50.637279 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:08:50.638767 20616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:08:50.640213 20616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:08:50.640290 20616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:50.640424 20616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:08:50.640479 20616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:08:50.680898 20616 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:40535
I20260925 11:08:50.680994 20743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:40535 every 8 connection(s)
I20260925 11:08:50.682230 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:08:50.688391 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 20616
I20260925 11:08:50.688582 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:08:50.689185 8916 external_mini_cluster.cc:1428] Setting key 30c928db0416421273574dd72faa9839
Sep 25 11:08:50 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334530, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:08:50.705922 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.685091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:38317 (local address 127.8.181.62:41339)
0925 11:08:50.685575 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.685580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.686728 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 11:08:50.688848 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.688853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.688943 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.689035 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:50.691204 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.692401 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.694847 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.695294 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.695402 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.699193 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:50.699198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:50.699202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:50.699239 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:50.701835 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.702594 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.702598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.702601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.702790 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.703404 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.703408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.703410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.703988 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:50.704116 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.705231 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.705511 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":274,"threads_started":1}
I20260925 11:08:50.706935 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.685632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41339 (local address 127.8.181.3:38317)
0925 11:08:50.686455 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.686566 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.688486 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.689346 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.689358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.689915 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:08:50.690987 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.691021 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.692602 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.692609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.694636 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.694663 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.695510 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.695515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.695741 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.696474 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:08:50.696499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:08:50.698950 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:08:50.701972 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:50.701980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:50.701993 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:50.702421 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:50.702887 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:08:50.702890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:08:50.702893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:08:50.703253 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:08:50.704238 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:08:50.704248 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:08:50.704688 (+ 440us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.705028 (+ 340us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.705565 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":332,"threads_started":1}
I20260925 11:08:50.708302 20744 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41339
I20260925 11:08:50.708637 20744 heartbeater.cc:464] Registering TS with master...
I20260925 11:08:50.709393 20744 heartbeater.cc:511] Master 127.8.181.62:41339 requested a full tablet report, sending...
I20260925 11:08:50.710932 20254 ts_manager.cc:194] Registered new tserver with Master: 8ed8a97a9dfb4d909cb7d76b4704ff1d (127.8.181.3:40535)
I20260925 11:08:50.711735 20254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:38317
I20260925 11:08:50.718750 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:08:50.738232 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.723883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.8.181.62:41339)
0925 11:08:50.724089 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.724096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.724629 (+ 533us) server_negotiation.cc:407] Connection header received
0925 11:08:50.724860 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.724865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.725346 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.725470 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:50.727282 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.728011 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.729387 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.729817 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.729963 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.732496 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:50.732501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:50.732504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:50.732537 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:50.735411 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.736161 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.736167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.736171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.736315 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:50.736717 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:50.736724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:50.736727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:50.736993 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:50.737265 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.737687 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.737892 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:08:50.742805 20254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:08:50.751013 20254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:08:50.788082 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.764856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.8.181.2:34473)
0925 11:08:50.777075 (+ 12219us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.777088 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.777721 (+ 633us) server_negotiation.cc:407] Connection header received
0925 11:08:50.778129 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.778142 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.778544 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.778789 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:08:50.781296 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.782697 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.785940 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.786959 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.787118 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.787323 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.787645 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.787782 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12034,"thread_start_us":11927,"threads_started":1}
I20260925 11:08:50.788780 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.764724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:34473 (local address 127.0.0.1:54726)
0925 11:08:50.777501 (+ 12777us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.777559 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.777815 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.779283 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.779291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.779327 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:08:50.781115 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.781122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.782864 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.782870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.785733 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.785752 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.787165 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.787171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.787377 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.787436 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.788070 (+ 634us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.788306 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12453,"mutex_wait_us":93,"thread_start_us":14025,"threads_started":1}
I20260925 11:08:50.789137 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.764896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41249 (local address 127.0.0.1:37920)
0925 11:08:50.779010 (+ 14114us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.779025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.779131 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.780115 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.780123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.780160 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:08:50.780775 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.780787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.782689 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.782694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.785762 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.785770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.788150 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.788159 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.788337 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.788385 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.788487 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.788608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14016,"mutex_wait_us":201,"thread_start_us":13200,"threads_started":1}
I20260925 11:08:50.789413 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.765548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.8.181.1:41249)
0925 11:08:50.777655 (+ 12107us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.777779 (+ 124us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.779142 (+ 1363us) server_negotiation.cc:407] Connection header received
0925 11:08:50.779333 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.779341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.779564 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.779733 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:08:50.780955 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.782538 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.785943 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.787395 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.787951 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.788209 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.788606 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.788783 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11940,"thread_start_us":11753,"threads_started":1}
I20260925 11:08:50.790093 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.779289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:40535 (local address 127.0.0.1:44470)
0925 11:08:50.780520 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.780539 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.780652 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.781506 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.781511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.781535 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:08:50.781932 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.781943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.783694 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.783699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.785335 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.785346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.789287 (+ 3941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.789299 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.789479 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.789511 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.789597 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.789721 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":6,"thread_start_us":957,"threads_started":1}
I20260925 11:08:50.790843 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.779562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.8.181.3:40535)
0925 11:08:50.780846 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.780852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.780907 (+ 55us) server_negotiation.cc:407] Connection header received
0925 11:08:50.780990 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.780997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.781302 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.781495 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:08:50.782121 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.783510 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.785501 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.786962 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.787105 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.787346 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.790321 (+ 2975us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.790528 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1125,"thread_start_us":988,"threads_started":1}
I20260925 11:08:50.793293 20545 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 (DEFAULT_TABLE table=test-table [id=46c0e8efbd1841caadb4d2af3598d445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:08:50.794236 20679 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 (DEFAULT_TABLE table=test-table [id=46c0e8efbd1841caadb4d2af3598d445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:08:50.794724 20545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf172e1279f48f2bf6483d38d7ccb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:08:50.795461 20679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf172e1279f48f2bf6483d38d7ccb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:08:50.797394 20402 tablet_service.cc:1467] Processing CreateTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 (DEFAULT_TABLE table=test-table [id=46c0e8efbd1841caadb4d2af3598d445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:08:50.799031 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bf172e1279f48f2bf6483d38d7ccb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:08:50.808465 20763 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Bootstrap starting.
I20260925 11:08:50.811352 20764 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Bootstrap starting.
I20260925 11:08:50.814581 20765 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Bootstrap starting.
I20260925 11:08:50.814610 20763 tablet_bootstrap.cc:654] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Neither blocks nor log segments found. Creating new log.
I20260925 11:08:50.815013 20764 tablet_bootstrap.cc:654] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Neither blocks nor log segments found. Creating new log.
I20260925 11:08:50.816592 20763 log.cc:824] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:08:50.816592 20764 log.cc:824] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Log is configured to *not* fsync() on all Append() calls
I20260925 11:08:50.818881 20765 tablet_bootstrap.cc:654] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Neither blocks nor log segments found. Creating new log.
I20260925 11:08:50.820429 20765 log.cc:824] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:08:50.821483 20763 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: No bootstrap required, opened a new log
I20260925 11:08:50.821580 20764 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: No bootstrap required, opened a new log
I20260925 11:08:50.821805 20763 ts_tablet_manager.cc:1397] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260925 11:08:50.822021 20764 ts_tablet_manager.cc:1397] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260925 11:08:50.824441 20765 tablet_bootstrap.cc:492] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: No bootstrap required, opened a new log
I20260925 11:08:50.824761 20765 ts_tablet_manager.cc:1397] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260925 11:08:50.830204 20764 raft_consensus.cc:348] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.830931 20764 raft_consensus.cc:374] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:08:50.831072 20764 raft_consensus.cc:729] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74573d0c40514c378564f08d44699c91, State: Initialized, Role: FOLLOWER
I20260925 11:08:50.831841 20764 consensus_queue.cc:260] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.833376 20610 heartbeater.cc:503] Master 127.8.181.62:41339 was elected leader, sending a full tablet report...
I20260925 11:08:50.834059 20764 ts_tablet_manager.cc:1428] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20260925 11:08:50.834340 20763 raft_consensus.cc:348] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.834874 20763 raft_consensus.cc:374] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:08:50.834990 20763 raft_consensus.cc:729] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed8a97a9dfb4d909cb7d76b4704ff1d, State: Initialized, Role: FOLLOWER
I20260925 11:08:50.835908 20763 consensus_queue.cc:260] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.837150 20744 heartbeater.cc:503] Master 127.8.181.62:41339 was elected leader, sending a full tablet report...
I20260925 11:08:50.837832 20763 ts_tablet_manager.cc:1428] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260925 11:08:50.838243 20765 raft_consensus.cc:348] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.838918 20765 raft_consensus.cc:374] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:08:50.839012 20765 raft_consensus.cc:729] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2e15bc1ba84cdab1ef67b2bf4bf50d, State: Initialized, Role: FOLLOWER
I20260925 11:08:50.839744 20765 consensus_queue.cc:260] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.841845 20765 ts_tablet_manager.cc:1428] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260925 11:08:50.843211 20611 tablet.cc:2369] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:08:50.879206 20769 raft_consensus.cc:482] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:08:50.879523 20769 raft_consensus.cc:504] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.881353 20769 leader_election.cc:288] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e2e15bc1ba84cdab1ef67b2bf4bf50d (127.8.181.1:41249), 8ed8a97a9dfb4d909cb7d76b4704ff1d (127.8.181.3:40535)
I20260925 11:08:50.889667 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.882035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35095 (local address 127.8.181.3:40535)
0925 11:08:50.882283 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.882293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.884196 (+ 1903us) server_negotiation.cc:407] Connection header received
0925 11:08:50.884283 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.884291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.884378 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.884458 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:08:50.885162 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.885908 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.887893 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.888686 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.888783 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.888924 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.889282 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.889404 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:08:50.889690 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.882040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:49903 (local address 127.8.181.1:41249)
0925 11:08:50.882244 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.882251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.882875 (+ 624us) server_negotiation.cc:407] Connection header received
0925 11:08:50.883110 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.883118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.883215 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.883316 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:08:50.884409 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.885608 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.887819 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.888705 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.888869 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.888969 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.889324 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.889447 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 11:08:50.889832 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.881732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:40535 (local address 127.8.181.2:35095)
0925 11:08:50.884048 (+ 2316us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.884065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.884167 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.884546 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.884550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.884578 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:08:50.884946 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.884958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.886073 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.886077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.887744 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.887762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.888920 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.888927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.889108 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.889147 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.889220 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.889373 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"mutex_wait_us":1,"thread_start_us":667,"threads_started":1}
I20260925 11:08:50.889930 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41249 (local address 127.8.181.2:49903)
0925 11:08:50.882758 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0925 11:08:50.882774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:08:50.882954 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:08:50.883369 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:08:50.883374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:08:50.883420 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:08:50.884231 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.884245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.885759 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.885764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.887654 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:08:50.887665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.888966 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.888975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:08:50.889112 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.889161 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 11:08:50.889273 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 11:08:50.889403 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":23,"thread_start_us":1572,"threads_started":1}
I20260925 11:08:50.891078 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf172e1279f48f2bf6483d38d7ccb77" candidate_uuid: "74573d0c40514c378564f08d44699c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" is_pre_election: true
I20260925 11:08:50.891078 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf172e1279f48f2bf6483d38d7ccb77" candidate_uuid: "74573d0c40514c378564f08d44699c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" is_pre_election: true
I20260925 11:08:50.891551 20422 raft_consensus.cc:2454] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74573d0c40514c378564f08d44699c91 in term 0.
I20260925 11:08:50.891559 20699 raft_consensus.cc:2454] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74573d0c40514c378564f08d44699c91 in term 0.
I20260925 11:08:50.892235 20498 leader_election.cc:302] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2e15bc1ba84cdab1ef67b2bf4bf50d, 74573d0c40514c378564f08d44699c91; no voters:
I20260925 11:08:50.892640 20769 raft_consensus.cc:2781] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:08:50.892796 20769 raft_consensus.cc:482] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:08:50.892908 20769 raft_consensus.cc:3037] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:08:50.895677 20769 raft_consensus.cc:504] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.896317 20769 leader_election.cc:288] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [CANDIDATE]: Term 1 election: Requested vote from peers 6e2e15bc1ba84cdab1ef67b2bf4bf50d (127.8.181.1:41249), 8ed8a97a9dfb4d909cb7d76b4704ff1d (127.8.181.3:40535)
I20260925 11:08:50.896944 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf172e1279f48f2bf6483d38d7ccb77" candidate_uuid: "74573d0c40514c378564f08d44699c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d"
I20260925 11:08:50.896972 20699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bf172e1279f48f2bf6483d38d7ccb77" candidate_uuid: "74573d0c40514c378564f08d44699c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d"
I20260925 11:08:50.897185 20699 raft_consensus.cc:3037] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:08:50.897185 20422 raft_consensus.cc:3037] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:08:50.900585 20699 raft_consensus.cc:2454] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74573d0c40514c378564f08d44699c91 in term 1.
I20260925 11:08:50.900585 20422 raft_consensus.cc:2454] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74573d0c40514c378564f08d44699c91 in term 1.
I20260925 11:08:50.901058 20498 leader_election.cc:302] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e2e15bc1ba84cdab1ef67b2bf4bf50d, 74573d0c40514c378564f08d44699c91, 8ed8a97a9dfb4d909cb7d76b4704ff1d; no voters:
I20260925 11:08:50.901403 20769 raft_consensus.cc:2781] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:08:50.901867 20769 raft_consensus.cc:686] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 1 LEADER]: Becoming Leader. State: Replica: 74573d0c40514c378564f08d44699c91, State: Running, Role: LEADER
I20260925 11:08:50.902420 20769 consensus_queue.cc:237] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } }
I20260925 11:08:50.907320 20254 catalog_manager.cc:5495] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74573d0c40514c378564f08d44699c91 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "74573d0c40514c378564f08d44699c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74573d0c40514c378564f08d44699c91" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 } health_report { overall_health: UNKNOWN } } }
W20260925 11:08:50.934233 20745 tablet.cc:2369] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:08:50.944128 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.937663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.8.181.2:34473)
0925 11:08:50.938009 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.938016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:50.938041 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:08:50.938273 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:50.938277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:50.938376 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:50.938568 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:08:50.939363 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.940065 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.941558 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:50.941971 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:50.942075 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:50.942689 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:08:50.943277 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:08:50.943403 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:50.943558 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 11:08:50.943762 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 11:08:50.953378 20699 raft_consensus.cc:1261] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 1 FOLLOWER]: Refusing update from remote peer 74573d0c40514c378564f08d44699c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:08:50.953379 20422 raft_consensus.cc:1261] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 1 FOLLOWER]: Refusing update from remote peer 74573d0c40514c378564f08d44699c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:08:50.954500 20774 consensus_queue.cc:1035] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed8a97a9dfb4d909cb7d76b4704ff1d" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:08:50.954905 20769 consensus_queue.cc:1035] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e2e15bc1ba84cdab1ef67b2bf4bf50d" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:08:50.967450 20777 mvcc.cc:205] Tried to move back new op lower bound from 7333210238773747712 to 7333210238580445184. Current Snapshot: MvccSnapshot[applied={T|T < 7333210238773747712}]
I20260925 11:08:50.982897 20254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:08:50.983189 20254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:08:50.986896 20254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 637f936006254b9dad2a4510fd513962: Sending DeleteTablet for 3 replicas of tablet 7bf172e1279f48f2bf6483d38d7ccb77
I20260925 11:08:50.988148 20679 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:08:50 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:44470
I20260925 11:08:50.988170 20402 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:08:50 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:37920
I20260925 11:08:50.988154 20545 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bf172e1279f48f2bf6483d38d7ccb77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:08:50 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:54726
I20260925 11:08:50.991282 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 20334
I20260925 11:08:51.021636 20789 tablet_replica.cc:330] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: stopping tablet replica
I20260925 11:08:51.022287 20789 raft_consensus.cc:2229] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:08:51.028213 20788 tablet_replica.cc:330] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: stopping tablet replica
I20260925 11:08:51.028505 20789 raft_consensus.cc:2258] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:08:51.028714 20788 raft_consensus.cc:2229] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:08:51.029263 20788 raft_consensus.cc:2258] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:08:51.029551 20787 tablet_replica.cc:330] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: stopping tablet replica
I20260925 11:08:51.029986 20787 raft_consensus.cc:2229] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:08:51.030720 20787 raft_consensus.cc:2258] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:08:51.031932 20788 ts_tablet_manager.cc:1905] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:08:51.032793 20789 ts_tablet_manager.cc:1905] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:08:51.032878 20787 ts_tablet_manager.cc:1905] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 11:08:51 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:08:51.040236 20788 ts_tablet_manager.cc:1918] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:08:51.040436 20788 log.cc:1196] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/7bf172e1279f48f2bf6483d38d7ccb77
I20260925 11:08:51.040591 20787 ts_tablet_manager.cc:1918] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:08:51.040781 20787 log.cc:1196] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/7bf172e1279f48f2bf6483d38d7ccb77
I20260925 11:08:51.041029 20788 ts_tablet_manager.cc:1939] T 7bf172e1279f48f2bf6483d38d7ccb77 P 6e2e15bc1ba84cdab1ef67b2bf4bf50d: Deleting consensus metadata
I20260925 11:08:51.041288 20787 ts_tablet_manager.cc:1939] T 7bf172e1279f48f2bf6483d38d7ccb77 P 74573d0c40514c378564f08d44699c91: Deleting consensus metadata
I20260925 11:08:51.043143 20238 catalog_manager.cc:4841] TS 6e2e15bc1ba84cdab1ef67b2bf4bf50d (127.8.181.1:41249): tablet 7bf172e1279f48f2bf6483d38d7ccb77 (table test-table [id=46c0e8efbd1841caadb4d2af3598d445]) successfully deleted
I20260925 11:08:51.043646 20240 catalog_manager.cc:4841] TS 74573d0c40514c378564f08d44699c91 (127.8.181.2:34473): tablet 7bf172e1279f48f2bf6483d38d7ccb77 (table test-table [id=46c0e8efbd1841caadb4d2af3598d445]) successfully deleted
I20260925 11:08:51.043888 20789 ts_tablet_manager.cc:1918] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:08:51.044060 20789 log.cc:1196] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/7bf172e1279f48f2bf6483d38d7ccb77
I20260925 11:08:51.044750 20789 ts_tablet_manager.cc:1939] T 7bf172e1279f48f2bf6483d38d7ccb77 P 8ed8a97a9dfb4d909cb7d76b4704ff1d: Deleting consensus metadata
I20260925 11:08:51.046370 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:50.993122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.8.181.1:41249)
0925 11:08:50.993314 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:50.993320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:51.017583 (+ 24263us) server_negotiation.cc:407] Connection header received
0925 11:08:51.018227 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:51.018236 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:51.018386 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:51.018545 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:51.022731 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.024372 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.029666 (+ 5294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.030402 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.030565 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:51.037021 (+ 6456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:51.037039 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:51.037058 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:51.037112 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:51.041989 (+ 4877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.043400 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.043411 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.043414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.043981 (+ 567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.044491 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.044499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.044506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.045020 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:51.045313 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:51.045773 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 11:08:51.046009 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:08:51.046844 20240 catalog_manager.cc:4841] TS 8ed8a97a9dfb4d909cb7d76b4704ff1d (127.8.181.3:40535): tablet 7bf172e1279f48f2bf6483d38d7ccb77 (table test-table [id=46c0e8efbd1841caadb4d2af3598d445]) successfully deleted
I20260925 11:08:51.047111 20462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:08:51.142000 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 20334
I20260925 11:08:51.163087 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 20473
Sep 25 11:08:51 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:08:51.180158 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:51.164041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.8.181.2:34473)
0925 11:08:51.164302 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:51.164308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:51.164329 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:08:51.164539 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:51.164543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:51.164616 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:51.164728 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:51.166002 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.166947 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.168263 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.168783 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.168943 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:51.173924 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:51.173941 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:51.173958 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:51.174006 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:51.177120 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.177896 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.177903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.177906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.178052 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.178461 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.178493 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.178500 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.178912 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:51.179259 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:51.179551 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 11:08:51.179779 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 11:08:51.180979 20605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:08:51.296551 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 20473
I20260925 11:08:51.315599 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 20616
Sep 25 11:08:51 dist-test-slave-6042 krb5kdc[19778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:08:51.332738 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:08:51.316605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.8.181.3:40535)
0925 11:08:51.316897 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 11:08:51.316902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:08:51.316922 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:08:51.316978 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:08:51.316981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:08:51.317046 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:08:51.317146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:08:51.318610 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.319361 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.320644 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:08:51.321032 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:08:51.321130 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:08:51.326063 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:08:51.326078 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:08:51.326096 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:08:51.326139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:08:51.329548 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.330421 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.330429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.330431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.330664 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:08:51.331229 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:08:51.331233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:08:51.331241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:08:51.331640 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:08:51.331948 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 11:08:51.332293 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 11:08:51.332477 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260925 11:08:51.333419 20739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:08:51.425184 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 20616
I20260925 11:08:51.442268 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 20222
I20260925 11:08:51.443588 20284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:08:51.559055 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 20222
I20260925 11:08:51.584122 8916 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 11:08:51.921128 8916 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 11:08:51.921406 8916 mini_ranger.cc:66] Stopping Ranger...
I20260925 11:08:52.268353 8916 mini_ranger.cc:68] Stopped Ranger
2026-09-25 11:08:52.268 UTC [19808] LOG: received smart shutdown request
2026-09-25 11:08:52.272 UTC [19808] LOG: background worker "logical replication launcher" (PID 19817) exited with exit code 1
2026-09-25 11:08:52.273 UTC [19812] LOG: shutting down
2026-09-25 11:08:52.487 UTC [19808] LOG: database system is shut down
2026-09-25T11:08:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54202 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:08:52 dist-test-slave-6042 krb5kdc[20803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:08:52 dist-test-slave-6042 krb5kdc[20803](info): set up 2 sockets
Sep 25 11:08:52 dist-test-slave-6042 krb5kdc[20803](info): commencing operation
krb5kdc: starting...
W20260925 11:08:53.767117 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.012s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:08:53 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:08:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangeradmin_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangeradmin_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangerlookup_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangerlookup_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
I20260925 11:08:54.122793 8916 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskA5RC6A/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/postgres -l logfile start
2026-09-25 11:08:57.002 UTC [20833] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 11:08:57.002 UTC [20833] LOG: listening on IPv4 address "127.8.181.20", port 49083
2026-09-25 11:08:57.007 UTC [20833] LOG: listening on Unix socket "/tmp/.s.PGSQL.49083"
2026-09-25 11:08:57.030 UTC [20836] LOG: database system was shut down at 2026-09-25 11:08:55 UTC
2026-09-25 11:08:57.034 UTC [20833] LOG: database system is ready to accept connections
I20260925 11:08:58.034935 8916 mini_postgres.cc:96] Postgres bound to 49083
2026-09-25 11:08:58.057 UTC [20844] FATAL: database "slave" does not exist
127.8.181.20:49083 - accepting connections
I20260925 11:08:58.057924 8916 mini_ranger.cc:161] Starting Ranger...
I20260925 11:08:58.087754 8916 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 11:08:58.674718 8916 mini_ranger.cc:87] Created ranger Postgres database
I20260925 11:08:58.674824 8916 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin
2026-09-25 11:08:58,981 [I] DB FLAVOR :POSTGRES
2026-09-25 11:08:58,981 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 11:08:58,982 [I] Checking connection..
2026-09-25 11:08:58,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 11:08:59,234 [I] Checking connection passed.
2026-09-25 11:08:59,234 [I] --------- Verifying version history table ---------
2026-09-25 11:08:59,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:08:59,489 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 11:08:59,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:08:59,761 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 11:08:59,761 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 11:08:59,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 11:09:00.002 UTC [20908] WARNING: there is no transaction in progress
2026-09-25 11:09:00,010 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 11:09:00,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 11:09:00,271 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 11:09:00,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 11:09:00,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 11:09:00,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 11:09:00,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6042.c.gcp-upstream.internal','N') ;"
2026-09-25 11:09:01,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 11:09:01,284 [I] Table x_portal_user does not exist in database ranger
2026-09-25 11:09:01,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 11:09:01,537 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 11:09:01,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 11:09:01,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 11:09:02,053 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 11:09:02,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 11:09:03.171 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.190 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.211 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.229 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.249 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.270 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.335 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.344 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.362 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.375 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.388 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.405 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.416 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.424 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.432 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:03.441 UTC [21016] WARNING: there is no transaction in progress
2026-09-25 11:09:04,246 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 11:09:04,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 11:09:04,509 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 11:09:04,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 11:09:04,759 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 11:09:04,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 11:09:05,025 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 11:09:05,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 11:09:05,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 11:09:05,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6042.c.gcp-upstream.internal';"
2026-09-25 11:09:05,802 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 11:09:05,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 11:09:06,079 [I] DB_PATCHES have already been applied
I20260925 11:09:06.085951 8916 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskA5RC6A/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 11:09:06.086050 8916 mini_ranger.cc:193] Using host: 127.8.181.20
I20260925 11:09:06.101954 8916 mini_ranger.cc:227] Ranger admin URL: http://127.8.181.20:48087
Sep 25, 2026 11:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 11:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 11:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 11:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 11:09:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 11:09:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 11:09:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 11:09:07.265349 8916 mini_ranger.cc:160] Time spent starting Ranger: real 9.207s user 0.000s sys 0.042s
Sep 25, 2026 11:09:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 11:09:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 11:09:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 11:09:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 11:09:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 11:09:30.512669 8916 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.181.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.181.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangerkms_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.181.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/rangerkms_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.181.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.181.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.181.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/HTTP_127.8.181.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 11:09:30 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334570, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 11:09:30.710841 8916 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 11:09:30.735131 8916 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 11:09:31.792375 8916 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 11:09:31.792501 8916 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms
I20260925 11:09:31.792548 8916 mini_ranger_kms.cc:227] Using postgres at 127.8.181.20:49083
2026-09-25 11:09:32,154 [I] DB FLAVOR :POSTGRES
2026-09-25 11:09:32,155 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 11:09:32,155 [I] Checking connection
2026-09-25 11:09:32,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 11:09:32,392 [I] connection success
2026-09-25 11:09:32,392 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 11:09:32,393 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 11:09:32,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 11:09:32,659 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 11:09:32,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 11:09:32,660 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 11:09:32,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.181.20:49083/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 11:09:32,971 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 11:09:33.485069 8916 mini_ranger_kms.cc:310] Created kms service
I20260925 11:09:33.701658 8916 mini_ranger_kms.cc:326] Created kudu user
I20260925 11:09:33.751251 8916 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 11:09:33 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334570, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:34.203778 8916 mini_ranger_kms.cc:384] Added ranger policy
I20260925 11:09:34.204106 8916 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms:/tmp/dist-test-taskA5RC6A/build/asan/bin/postgresql.jar:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskA5RC6A/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 11:09:34.204186 8916 mini_ranger_kms.cc:240] Using host: 127.8.181.20
I20260925 11:09:34.222350 8916 mini_ranger_kms.cc:276] Ranger KMS PID: 21197
I20260925 11:09:34.222506 8916 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.181.20:37795
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 11:09:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 11:09:36.517484 8916 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.807s user 0.049s sys 0.331s
I20260925 11:09:36.517757 8916 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 11:09:36.517956 8916 mini_ranger_kms.cc:401] 127.8.181.20:37795/kms/v1/keys
Sep 25 11:09:36 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790334576, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 11:09:36 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334576, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
Sep 25 11:09:37 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334576, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.181.20@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
Sep 25 11:09:39 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:09:39.711174 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38197
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:38197
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:40.058372 21247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:40.058758 21247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:40.058911 21247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:40.066798 21247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:40.066926 21247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:40.067008 21247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:40.067121 21247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:40.067189 21247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:40.077713 21247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.181.20:37795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:38197
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:40.079131 21247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:40.080492 21247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:40.093431 21253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:40.093736 21247 server_base.cc:1034] running on GCE node
W20260925 11:09:40.093434 21255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:40.093434 21252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:40.095000 21247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:40.096400 21247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:40.097613 21247 hybrid_clock.cc:648] HybridClock initialized: now 1790334580097560 us; error 52 us; skew 500 ppm
I20260925 11:09:40.097975 21247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:40 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334580, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:40.108737 21247 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:09:40.110524 21247 webserver.cc:460] Webserver started at http://127.8.181.62:42817/ using document root <none> and password file <none>
I20260925 11:09:40.111274 21247 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:40.111405 21247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:40.111749 21247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:09:40 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334580, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:40.162761 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "d9122c816fdf444aa5ca336ce5dfc83a"
format_stamp: "Formatted at 2026-09-25 11:09:40 on dist-test-slave-6042"
server_key: "58aedf729d0967c0ba96f5b6ffc327e9"
server_key_iv: "2f281c218f5b449dbe923def82e4b9f0"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:40.163677 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "d9122c816fdf444aa5ca336ce5dfc83a"
format_stamp: "Formatted at 2026-09-25 11:09:40 on dist-test-slave-6042"
server_key: "58aedf729d0967c0ba96f5b6ffc327e9"
server_key_iv: "2f281c218f5b449dbe923def82e4b9f0"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:40.169836 21247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260925 11:09:40.186967 21262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:40.188524 21247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:09:40.188735 21247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d9122c816fdf444aa5ca336ce5dfc83a"
format_stamp: "Formatted at 2026-09-25 11:09:40 on dist-test-slave-6042"
server_key: "58aedf729d0967c0ba96f5b6ffc327e9"
server_key_iv: "2f281c218f5b449dbe923def82e4b9f0"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:40.188966 21247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:40.207978 21247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:40.211824 21247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:40.212138 21247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:40.231354 21247 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:38197
I20260925 11:09:40.231377 21313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:38197 every 8 connection(s)
I20260925 11:09:40.232802 21247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:09:40.236356 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21247
I20260925 11:09:40.236582 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:09:40.237172 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:40.237470 8916 external_mini_cluster.cc:1428] Setting key 7284f558b7234dea90bcdf9cd5e90dc3
I20260925 11:09:40.250351 21314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Bootstrap starting.
Sep 25 11:09:40 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:40.253527 21314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:40.254894 21314 log.cc:824] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:40.258339 21314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: No bootstrap required, opened a new log
I20260925 11:09:40.261817 21317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:40.239643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.8.181.62:38197)
0925 11:09:40.240442 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:40.240454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:40.240495 (+ 41us) server_negotiation.cc:407] Connection header received
0925 11:09:40.242559 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:40.242584 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:40.243079 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:40.243532 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:40.245020 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:40.246530 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:40.247979 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:40.248767 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:40.248992 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:40.253370 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:40.253379 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:40.253394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:40.253451 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:40.257331 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:40.258124 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:40.258139 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:40.258146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:40.258311 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:40.258786 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:40.258798 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:40.258806 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:40.259358 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:40.259652 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:40.260214 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 11:09:40.260606 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":366,"threads_started":1}
I20260925 11:09:40.266697 21314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:40.267112 21314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:40.267249 21314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9122c816fdf444aa5ca336ce5dfc83a, State: Initialized, Role: FOLLOWER
I20260925 11:09:40.267800 21314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:40.268038 21314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:09:40.268183 21314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:09:40.268350 21314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:40.270541 21314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:40.271087 21314 leader_election.cc:302] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9122c816fdf444aa5ca336ce5dfc83a; no voters:
I20260925 11:09:40.271739 21314 leader_election.cc:288] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:09:40.271864 21319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:09:40.272593 21319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 LEADER]: Becoming Leader. State: Replica: d9122c816fdf444aa5ca336ce5dfc83a, State: Running, Role: LEADER
I20260925 11:09:40.273224 21319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:40.273830 21314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:40.276341 21320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } } }
I20260925 11:09:40.276790 21320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:40.277387 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9122c816fdf444aa5ca336ce5dfc83a. Latest consensus state: current_term: 1 leader_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } } }
I20260925 11:09:40.277748 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:40.277801 21324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:09:40.285387 21324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:09:40.291863 21314 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 11:09:40.303361 21324 catalog_manager.cc:1347] Generated new cluster ID: 59d1c26c87fc46fea8c5ed83002fa38a
I20260925 11:09:40.303582 21324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:09:40.321534 21324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:09:40.323920 21324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:09:40.341365 21324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Generated new TSK 0
I20260925 11:09:40.342648 21324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:09:41.170977 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:09:41 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 11:09:41.659706 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:41.660235 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:41.660458 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:41.673843 21359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:41.674024 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:41.674290 21359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:09:41.692371 21359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:41.694126 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:41.695927 21359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:41.723830 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:41.724602 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:41.729574 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:41.731316 21359 server_base.cc:1034] running on GCE node
I20260925 11:09:41.732241 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:41.733430 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:41.734642 21359 hybrid_clock.cc:648] HybridClock initialized: now 1790334581734562 us; error 66 us; skew 500 ppm
I20260925 11:09:41.734995 21359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:41 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:41.739593 21359 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:09:41.746033 21359 webserver.cc:460] Webserver started at http://127.8.181.1:34853/ using document root <none> and password file <none>
I20260925 11:09:41.747112 21359 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:41.747280 21359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:41.747661 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:09:41 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:41.766010 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "955e4e298e3e4a6a8177cdd46eb3672f"
format_stamp: "Formatted at 2026-09-25 11:09:41 on dist-test-slave-6042"
server_key: "c697dd82c608e14d49caaa3f46be43e6"
server_key_iv: "ff95231367fac0694e9f4c1ff0bd1e93"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:41.766986 21359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "955e4e298e3e4a6a8177cdd46eb3672f"
format_stamp: "Formatted at 2026-09-25 11:09:41 on dist-test-slave-6042"
server_key: "c697dd82c608e14d49caaa3f46be43e6"
server_key_iv: "ff95231367fac0694e9f4c1ff0bd1e93"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:41.775447 21359 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260925 11:09:41.804179 21379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:41.806138 21359 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.004s
I20260925 11:09:41.806417 21359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "955e4e298e3e4a6a8177cdd46eb3672f"
format_stamp: "Formatted at 2026-09-25 11:09:41 on dist-test-slave-6042"
server_key: "c697dd82c608e14d49caaa3f46be43e6"
server_key_iv: "ff95231367fac0694e9f4c1ff0bd1e93"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:41.806807 21359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:41.843462 21359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:41.849043 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:41.849442 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:41.856870 21359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:41.864923 21359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:41.865123 21359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 11:09:41.865353 21359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:41.865433 21359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:41.930357 21359 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:41379
I20260925 11:09:41.930373 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:41379 every 8 connection(s)
I20260925 11:09:41.932188 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:09:41.939872 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21359
I20260925 11:09:41.940145 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:09:41.941164 8916 external_mini_cluster.cc:1428] Setting key ecbdf7a8ec22cb6763e080156c9469cc
Sep 25 11:09:41 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:41.993418 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:41.950715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:38097 (local address 127.8.181.62:38197)
0925 11:09:41.967384 (+ 16669us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:41.967391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:41.967418 (+ 27us) server_negotiation.cc:407] Connection header received
0925 11:09:41.969015 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:41.969021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:41.969112 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:41.969283 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:41.971193 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:41.972416 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:41.975005 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:41.975653 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:41.975785 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:41.984123 (+ 8338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:41.984131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:41.984135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:41.984182 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:41.987901 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:41.989032 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:41.989038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:41.989042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:41.989212 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:41.989766 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:41.989772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:41.989776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:41.990135 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:41.990292 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:41.992784 (+ 2492us) server_negotiation.cc:299] Negotiation successful
0925 11:09:41.993038 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16496,"thread_start_us":16669,"threads_started":1}
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:09:41.994815 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:41.945362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.1:38097)
0925 11:09:41.966367 (+ 21005us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:41.966430 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:41.968679 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:41.969550 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:41.969560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:41.970032 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:41.970950 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:41.970980 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:41.972586 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:41.972592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:41.974812 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:41.974842 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:41.976089 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:41.976096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:41.976414 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:41.977251 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:41.977296 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:41.983781 (+ 6485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:41.988110 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:41.988121 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:41.988141 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:41.988871 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:41.989369 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:41.989374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:41.989379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:41.989630 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:41.990377 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:41.990386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:41.990894 (+ 508us) client_negotiation.cc:769] Sending connection context
0925 11:09:41.991195 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 11:09:41.991761 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20670,"thread_start_us":20427,"threads_started":1}
I20260925 11:09:41.996968 21492 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:41.997313 21492 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:41.998198 21492 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:42.000926 21279 ts_manager.cc:194] Registered new tserver with Master: 955e4e298e3e4a6a8177cdd46eb3672f (127.8.181.1:41379)
I20260925 11:09:42.003034 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:38097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Sep 25 11:09:42 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
W20260925 11:09:42.046535 8916 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.181.2: real 0.102s user 0.001s sys 0.018s
I20260925 11:09:42.047580 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:09:42 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334581, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
W20260925 11:09:42.503229 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:42.503759 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:42.503969 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:42.515904 21498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:42.516151 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:42.516441 21498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:09:42.533476 21498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:42.535532 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:42.537380 21498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:42.575013 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:42.577576 21509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:42.578090 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:42.578828 21498 server_base.cc:1034] running on GCE node
I20260925 11:09:42.581606 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:42.595244 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:42.598747 21498 hybrid_clock.cc:648] HybridClock initialized: now 1790334582598613 us; error 125 us; skew 500 ppm
I20260925 11:09:42.599237 21498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:42 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:42.603703 21498 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:09:42.638957 21498 webserver.cc:460] Webserver started at http://127.8.181.2:40635/ using document root <none> and password file <none>
I20260925 11:09:42.640206 21498 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:42.640467 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:42.640892 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:09:42 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:42.674153 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "431a79cc568340e3af2e993a2c3ad3ba"
format_stamp: "Formatted at 2026-09-25 11:09:42 on dist-test-slave-6042"
server_key: "04694a5ba66a772491e64359774203a8"
server_key_iv: "b03b9189228143abffaa35ca18d67448"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:42.675401 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "431a79cc568340e3af2e993a2c3ad3ba"
format_stamp: "Formatted at 2026-09-25 11:09:42 on dist-test-slave-6042"
server_key: "04694a5ba66a772491e64359774203a8"
server_key_iv: "b03b9189228143abffaa35ca18d67448"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:42.695667 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.000s sys 0.015s
I20260925 11:09:42.750391 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:42.752496 21498 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.004s sys 0.002s
I20260925 11:09:42.752804 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "431a79cc568340e3af2e993a2c3ad3ba"
format_stamp: "Formatted at 2026-09-25 11:09:42 on dist-test-slave-6042"
server_key: "04694a5ba66a772491e64359774203a8"
server_key_iv: "b03b9189228143abffaa35ca18d67448"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:42.753159 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:42.807410 21498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:42.812935 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:42.813334 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:42.826912 21498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:42.829907 21498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:42.830015 21498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:42.830125 21498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:42.830186 21498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:42.871688 21498 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44379
I20260925 11:09:42.871711 21634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44379 every 8 connection(s)
I20260925 11:09:42.873371 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:09:42.882622 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21498
I20260925 11:09:42.882786 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:09:42.883351 8916 external_mini_cluster.cc:1428] Setting key 2e4360718c405d0ebbcc69735d682982
Sep 25 11:09:42 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:09:42.914896 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:42.876268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:59811 (local address 127.8.181.62:38197)
0925 11:09:42.891601 (+ 15333us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:42.891608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:42.892390 (+ 782us) server_negotiation.cc:407] Connection header received
0925 11:09:42.895338 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:42.895344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:42.895432 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:42.895523 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:42.897875 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:42.898953 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:42.901755 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:42.902366 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:42.902505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:42.907459 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:42.907466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:42.907471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:42.907520 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:42.911010 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:42.912072 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:42.912079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:42.912082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:42.912244 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:42.912780 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:42.912785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:42.912788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:42.913144 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:42.913278 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:42.914261 (+ 983us) server_negotiation.cc:299] Negotiation successful
0925 11:09:42.914530 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15190,"thread_start_us":19413,"threads_started":1}
I20260925 11:09:42.916107 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:42.877063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.2:59811)
0925 11:09:42.892199 (+ 15136us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:42.892257 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:42.894898 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:42.896115 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:42.896127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:42.896784 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:42.897603 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:42.897683 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:42.899214 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:42.899234 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:42.901555 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:42.901574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:42.902870 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:42.902877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:42.903175 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:42.904080 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:42.904111 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:42.907074 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:42.911280 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:42.911290 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:42.911312 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:42.911917 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:42.912391 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:42.912395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:42.912398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:42.912649 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:42.913380 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:42.913388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:42.913826 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 11:09:42.914143 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 11:09:42.914746 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14790,"thread_start_us":18531,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
I20260925 11:09:42.918136 21635 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:42.918494 21635 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:42.919360 21635 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:42.921170 21279 ts_manager.cc:194] Registered new tserver with Master: 431a79cc568340e3af2e993a2c3ad3ba (127.8.181.2:44379)
I20260925 11:09:42.922299 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:59811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:09:42.958755 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:09:43.007612 21492 heartbeater.cc:503] Master 127.8.181.62:38197 was elected leader, sending a full tablet report...
W20260925 11:09:43.367326 21641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:43.367650 21641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:43.367810 21641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:43.375646 21641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:43.375767 21641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:43.375931 21641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:09:43.386113 21641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:43.387465 21641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:43.388887 21641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:43.403802 21646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:43.403829 21647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:43.404104 21649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:43.405572 21641 server_base.cc:1034] running on GCE node
I20260925 11:09:43.406337 21641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:43.407500 21641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:43.408723 21641 hybrid_clock.cc:648] HybridClock initialized: now 1790334583408649 us; error 70 us; skew 500 ppm
I20260925 11:09:43.409060 21641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:43.412528 21641 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:09:43.414198 21641 webserver.cc:460] Webserver started at http://127.8.181.3:39533/ using document root <none> and password file <none>
I20260925 11:09:43.414980 21641 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:43.415125 21641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:43.415432 21641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:43.434695 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "2cc84c780a9044ce84319857afc00d13"
format_stamp: "Formatted at 2026-09-25 11:09:43 on dist-test-slave-6042"
server_key: "0408680225d8cfb12fae451760342640"
server_key_iv: "ac3766837b02c1f36d0970cb9610988e"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:43.436527 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "2cc84c780a9044ce84319857afc00d13"
format_stamp: "Formatted at 2026-09-25 11:09:43 on dist-test-slave-6042"
server_key: "0408680225d8cfb12fae451760342640"
server_key_iv: "ac3766837b02c1f36d0970cb9610988e"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:43.442997 21641 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 11:09:43.466722 21656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:43.468815 21641 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.007s sys 0.000s
I20260925 11:09:43.469105 21641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "2cc84c780a9044ce84319857afc00d13"
format_stamp: "Formatted at 2026-09-25 11:09:43 on dist-test-slave-6042"
server_key: "0408680225d8cfb12fae451760342640"
server_key_iv: "ac3766837b02c1f36d0970cb9610988e"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:43.469451 21641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:43.504112 21641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:43.508664 21641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:43.508999 21641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:43.510074 21641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:43.511726 21641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:43.511837 21641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:43.511994 21641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:43.512076 21641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:43.559406 21641 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:44369
I20260925 11:09:43.559422 21768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:44369 every 8 connection(s)
I20260925 11:09:43.560716 21641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:09:43.561617 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21641
I20260925 11:09:43.561859 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:09:43.562636 8916 external_mini_cluster.cc:1428] Setting key 2e2242280ff2e59b05846f3d4a1e0c6a
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:43.591717 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:43.563701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:32985 (local address 127.8.181.62:38197)
0925 11:09:43.571001 (+ 7300us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:43.571007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:43.575328 (+ 4321us) server_negotiation.cc:407] Connection header received
0925 11:09:43.577342 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:43.577348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:43.577456 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:43.577584 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:43.579136 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.579876 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.581594 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.582040 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.582174 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:43.586245 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:43.586251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:43.586256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:43.586305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:43.588848 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.589662 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.589667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.589669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.589796 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.590179 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.590182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.590184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.590431 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:43.590640 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:43.591265 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 11:09:43.591465 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7167,"thread_start_us":6967,"threads_started":1}
I20260925 11:09:43.592754 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:43.564303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.3:32985)
0925 11:09:43.575161 (+ 10858us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:43.575216 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:43.577039 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:43.577840 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:43.577850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:43.578289 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:43.578965 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:43.578983 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.580040 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.580044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:43.581444 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:43.581462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.582273 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.582280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:43.582590 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:43.583289 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:43.583312 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:43.585986 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:43.588994 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:43.589001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:43.589018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:43.589549 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:43.589886 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:43.589890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:43.589892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:43.590082 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:43.590616 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:43.590622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:43.590946 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 11:09:43.591178 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 11:09:43.591581 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10537,"thread_start_us":10321,"threads_started":1}
I20260925 11:09:43.594170 21769 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:43.594566 21769 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:43.595304 21769 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:43.596942 21279 ts_manager.cc:194] Registered new tserver with Master: 2cc84c780a9044ce84319857afc00d13 (127.8.181.3:44369)
I20260925 11:09:43.598189 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:32985
I20260925 11:09:43.599285 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:09:43.601660 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21359
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:09:43.619578 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:43.602724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.8.181.1:41379)
0925 11:09:43.603428 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:43.603435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:43.603458 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:09:43.603513 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:43.603519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:43.603708 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:43.603884 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:43.605403 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.606403 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.607942 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.608510 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.608655 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:43.613486 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:43.613499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:43.613513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:43.613554 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:43.616517 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.617332 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.617340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.617342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.617504 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.617893 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.617899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.617902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.618252 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:43.618606 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:43.618919 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 11:09:43.619189 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260925 11:09:43.620972 21487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:43.706522 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21359
I20260925 11:09:43.723562 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21498
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:09:43.739985 21776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:43.724458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.8.181.2:44379)
0925 11:09:43.724970 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:43.724976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:43.725001 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:09:43.725060 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:43.725067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:43.725270 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:43.725394 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:43.726830 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.727834 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.729252 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.729794 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.729898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:43.734358 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:43.734370 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:43.734388 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:43.734431 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:43.737343 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.738038 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.738044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.738046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.738177 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.738527 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.738537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.738539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.738860 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:43.739073 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:43.739366 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 11:09:43.739561 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260925 11:09:43.741317 21630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:43.824471 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21498
I20260925 11:09:43.840106 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21641
Sep 25 11:09:43 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:09:43.864678 21778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:43.841238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.8.181.3:44369)
0925 11:09:43.841910 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:43.841918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:43.841949 (+ 31us) server_negotiation.cc:407] Connection header received
0925 11:09:43.842016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:43.842023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:43.842230 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:43.842368 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:43.843794 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.844860 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.846075 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:43.846741 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:43.846843 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:43.858391 (+ 11548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:43.858408 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:43.858426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:43.858584 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:43.861533 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.862415 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.862422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.862424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.862616 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:43.863018 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:43.863025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:43.863028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:43.863388 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:43.863726 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:43.864041 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 11:09:43.864303 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":300,"threads_started":1}
I20260925 11:09:43.866079 21764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:43.954078 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21641
I20260925 11:09:43.975641 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21247
I20260925 11:09:43.977191 21309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:44.090245 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21247
I20260925 11:09:44.114223 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38197
--webserver_interface=127.8.181.62
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:38197
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:44.452590 21781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:44.452940 21781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:44.453101 21781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:44.460681 21781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:44.460803 21781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:44.460848 21781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:44.460947 21781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:44.461033 21781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:44.471910 21781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:38197
--ranger_config_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:44.473294 21781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:44.474735 21781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:44.486907 21789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:44.486991 21787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:44.487001 21786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:44.487834 21781 server_base.cc:1034] running on GCE node
I20260925 11:09:44.488636 21781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:44.490082 21781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:44.491281 21781 hybrid_clock.cc:648] HybridClock initialized: now 1790334584491245 us; error 37 us; skew 500 ppm
I20260925 11:09:44.491647 21781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:44 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334584, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:44.494801 21781 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:09:44.496404 21781 webserver.cc:460] Webserver started at http://127.8.181.62:42817/ using document root <none> and password file <none>
I20260925 11:09:44.497095 21781 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:44.497244 21781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 11:09:44 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334584, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:44.555516 21781 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:09:44.558965 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:44.560566 21781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:09:44.560856 21781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d9122c816fdf444aa5ca336ce5dfc83a"
format_stamp: "Formatted at 2026-09-25 11:09:40 on dist-test-slave-6042"
server_key: "58aedf729d0967c0ba96f5b6ffc327e9"
server_key_iv: "2f281c218f5b449dbe923def82e4b9f0"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:44.561945 21781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:44.585333 21781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:44.590102 21781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:44.590425 21781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:44.610348 21781 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:38197
I20260925 11:09:44.610371 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:38197 every 8 connection(s)
I20260925 11:09:44.611984 21781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:09:44.619336 21849 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:09:44.620222 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21781
I20260925 11:09:44.621376 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:41379
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:38197
--webserver_port=34853
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:09:44.623046 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Bootstrap starting.
I20260925 11:09:44.642737 21849 log.cc:824] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:44.651952 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:09:44.652504 21849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Bootstrap complete.
I20260925 11:09:44.661760 21849 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:44.662756 21849 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9122c816fdf444aa5ca336ce5dfc83a, State: Initialized, Role: FOLLOWER
I20260925 11:09:44.663443 21849 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:44.663686 21849 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:09:44.663806 21849 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:09:44.663998 21849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:09:44.668356 21849 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:44.668915 21849 leader_election.cc:302] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9122c816fdf444aa5ca336ce5dfc83a; no voters:
I20260925 11:09:44.669550 21849 leader_election.cc:288] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 11:09:44.669698 21853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 11:09:44.672116 21849 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:44.672616 21853 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [term 2 LEADER]: Becoming Leader. State: Replica: d9122c816fdf444aa5ca336ce5dfc83a, State: Running, Role: LEADER
I20260925 11:09:44.673233 21853 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } }
I20260925 11:09:44.676157 21855 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9122c816fdf444aa5ca336ce5dfc83a. Latest consensus state: current_term: 2 leader_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } } }
I20260925 11:09:44.676523 21855 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:44.676970 21854 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9122c816fdf444aa5ca336ce5dfc83a" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38197 } } }
I20260925 11:09:44.677362 21854 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:44.691660 21862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:09:44.697458 21862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:09:44.698886 21862 catalog_manager.cc:1259] Loaded cluster ID: 59d1c26c87fc46fea8c5ed83002fa38a
I20260925 11:09:44.698992 21862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:09:44.703353 21862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:09:44.704838 21862 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Loaded TSK: 0
I20260925 11:09:44.706358 21862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:09:44.709095 21849 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 11:09:45.059631 21851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:45.060062 21851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:45.060215 21851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:45.072033 21851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:45.072188 21851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:45.072379 21851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:09:45.086144 21851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:41379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=34853
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:45.087540 21851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:45.089087 21851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:45.120663 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:45.121397 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:45.126168 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:45.126806 21851 server_base.cc:1034] running on GCE node
I20260925 11:09:45.127660 21851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:45.129212 21851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:45.132550 21851 hybrid_clock.cc:648] HybridClock initialized: now 1790334585132455 us; error 70 us; skew 500 ppm
I20260925 11:09:45.132983 21851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:45 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:45.142828 21851 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:09:45.154233 21851 webserver.cc:460] Webserver started at http://127.8.181.1:34853/ using document root <none> and password file <none>
I20260925 11:09:45.155189 21851 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:45.155388 21851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 11:09:45 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:45.188968 21851 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.007s
I20260925 11:09:45.218662 21897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:45.221055 21851 fs_manager.cc:725] Time spent opening block manager: real 0.030s user 0.011s sys 0.017s
I20260925 11:09:45.221383 21851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "955e4e298e3e4a6a8177cdd46eb3672f"
format_stamp: "Formatted at 2026-09-25 11:09:41 on dist-test-slave-6042"
server_key: "c697dd82c608e14d49caaa3f46be43e6"
server_key_iv: "ff95231367fac0694e9f4c1ff0bd1e93"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:45.223027 21851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:45.277753 21851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:45.284090 21851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:45.284499 21851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:45.292114 21851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:45.304440 21851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:45.304670 21851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:45.304863 21851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:45.305001 21851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:45.374661 21851 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:41379
I20260925 11:09:45.374696 22010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:41379 every 8 connection(s)
I20260925 11:09:45.376478 21851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:09:45.381578 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 21851
I20260925 11:09:45.382532 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:44379
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:38197
--webserver_port=40635
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:09:45 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:45.417058 22014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:45.380783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.1:44995)
0925 11:09:45.395264 (+ 14481us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:45.395337 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:45.398056 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:45.400390 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:45.400398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:45.400851 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:45.401409 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:45.401426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:45.403305 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:45.403318 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:45.404574 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:45.404598 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:45.405637 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:45.405644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:45.405922 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:45.406678 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:45.406701 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:45.408794 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:45.412416 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:45.412425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:45.412441 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:45.412873 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:45.413326 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:45.413331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:45.413334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:45.413550 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:45.414448 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:45.414456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:45.414808 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 11:09:45.415090 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 11:09:45.415677 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14055,"thread_start_us":18274,"threads_started":1}
I20260925 11:09:45.417058 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:45.380531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:44995 (local address 127.8.181.62:38197)
0925 11:09:45.385243 (+ 4712us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:45.385272 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:45.398644 (+ 13372us) server_negotiation.cc:407] Connection header received
0925 11:09:45.399210 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:45.399230 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:45.399783 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:45.400219 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:45.401646 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:45.403137 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:45.404751 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:45.405399 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:45.405603 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:45.409056 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:45.409064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:45.409079 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:45.409120 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:45.412256 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:45.413036 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:45.413043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:45.413047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:45.413200 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:45.413680 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:45.413691 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:45.413693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:45.414288 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:45.414523 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:45.415297 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 11:09:45.415671 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4253,"thread_start_us":9776,"threads_started":1}
I20260925 11:09:45.421345 22011 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:45.421693 22011 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:45.422634 22011 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
W20260925 11:09:45.425966 22011 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:38197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 11:09:45.426179 22011 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:45.426533 22011 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:45.427707 22011 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:45.427965 22011 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
W20260925 11:09:45.429064 22011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:09:45.837263 22015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:45.837649 22015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:45.837807 22015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:45.848075 22015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:45.848269 22015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:45.848477 22015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:09:45.864868 22015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:44379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=40635
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:45.866773 22015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:45.868506 22015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:45.891335 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:45.891361 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:45.891512 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:45.892097 22015 server_base.cc:1034] running on GCE node
I20260925 11:09:45.892755 22015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:45.893931 22015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:45.895153 22015 hybrid_clock.cc:648] HybridClock initialized: now 1790334585895095 us; error 48 us; skew 500 ppm
I20260925 11:09:45.895498 22015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:45 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:45.898865 22015 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:09:45.900542 22015 webserver.cc:460] Webserver started at http://127.8.181.2:40635/ using document root <none> and password file <none>
I20260925 11:09:45.901329 22015 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:45.901479 22015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 11:09:45 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:45.947507 22015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 11:09:45.950695 22041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:45.952654 22015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 11:09:45.952850 22015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "431a79cc568340e3af2e993a2c3ad3ba"
format_stamp: "Formatted at 2026-09-25 11:09:42 on dist-test-slave-6042"
server_key: "04694a5ba66a772491e64359774203a8"
server_key_iv: "b03b9189228143abffaa35ca18d67448"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:45.954258 22015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:46.015777 22015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:46.021494 22015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:46.021836 22015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:46.023550 22015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:46.025477 22015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:46.025590 22015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:46.025702 22015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:46.025771 22015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:46.109195 22015 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44379
I20260925 11:09:46.109244 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44379 every 8 connection(s)
I20260925 11:09:46.110759 22015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:09:46.120934 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22015
I20260925 11:09:46.121752 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.181.20:37795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:44369
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:38197
--webserver_port=39533
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334585, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:46.136634 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.113993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.2:36929)
0925 11:09:46.114797 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:46.114853 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:46.116527 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:46.117315 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:46.117324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:46.117765 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:46.118350 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.118401 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.120064 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.120070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.122103 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.122119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.123402 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.123409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.123690 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.124509 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:46.124539 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:46.127370 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:46.131221 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:46.131231 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:46.131255 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:46.131915 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:46.132389 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:46.132397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:46.132403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:46.132669 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:46.133369 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:46.133381 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:46.133861 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 11:09:46.134170 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 11:09:46.135117 (+ 947us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":331,"threads_started":1}
I20260925 11:09:46.138514 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.113448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:36929 (local address 127.8.181.62:38197)
0925 11:09:46.113952 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.113959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.114995 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 11:09:46.116824 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.116830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.116930 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.117055 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:46.118652 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.119840 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.122293 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.122973 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.123099 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.127631 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:46.127636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:46.127639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:46.127674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:46.131018 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.132088 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.132096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.132102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.132254 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.132817 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.132825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.132831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.133213 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:46.137554 (+ 4341us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.137729 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.138013 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260925 11:09:46.139858 22155 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:46.140187 22155 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:46.141070 22155 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:46.143229 21814 ts_manager.cc:194] Registered new tserver with Master: 431a79cc568340e3af2e993a2c3ad3ba (127.8.181.2:44379)
I20260925 11:09:46.145700 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:36929
I20260925 11:09:46.429200 22011 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:46.429661 22011 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:46.431110 21814 ts_manager.cc:194] Registered new tserver with Master: 955e4e298e3e4a6a8177cdd46eb3672f (127.8.181.1:41379)
I20260925 11:09:46.432332 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:44995
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
W20260925 11:09:46.550076 22159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:46.550525 22159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:46.550732 22159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:46.561784 22159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:46.561935 22159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:46.562139 22159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:09:46.574234 22159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.181.20:37795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:44369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=39533
--tserver_master_addrs=127.8.181.62:38197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:46.576236 22159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:46.578261 22159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:46.623574 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:46.625201 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:46.639464 22169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:46.640007 22159 server_base.cc:1034] running on GCE node
I20260925 11:09:46.640969 22159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:46.647228 22159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:46.648476 22159 hybrid_clock.cc:648] HybridClock initialized: now 1790334586648400 us; error 81 us; skew 500 ppm
I20260925 11:09:46.648890 22159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:46.653168 22159 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:09:46.662158 22159 webserver.cc:460] Webserver started at http://127.8.181.3:39533/ using document root <none> and password file <none>
I20260925 11:09:46.663353 22159 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:46.663589 22159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:46.700763 22159 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260925 11:09:46.704852 22176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:46.706965 22159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:09:46.707147 22159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "2cc84c780a9044ce84319857afc00d13"
format_stamp: "Formatted at 2026-09-25 11:09:43 on dist-test-slave-6042"
server_key: "0408680225d8cfb12fae451760342640"
server_key_iv: "ac3766837b02c1f36d0970cb9610988e"
server_key_version: "kuduclusterkey@0"
I20260925 11:09:46.708550 22159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:46.739228 22159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:46.744431 22159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:46.744771 22159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:46.746050 22159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:46.747920 22159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:46.748028 22159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:46.748131 22159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:46.748206 22159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:46.790407 22159 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:44369
I20260925 11:09:46.790484 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:44369 every 8 connection(s)
I20260925 11:09:46.792285 22159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:09:46.798182 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22159
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:46.834230 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.795060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:53653 (local address 127.8.181.62:38197)
0925 11:09:46.806912 (+ 11852us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.806922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.811545 (+ 4623us) server_negotiation.cc:407] Connection header received
0925 11:09:46.815263 (+ 3718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.815269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.815360 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.815432 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:46.817763 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.819012 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.821623 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.822325 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.822409 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.827428 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:46.827435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:46.827439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:46.827485 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:46.830574 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.831578 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.831583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.831586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.831725 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.832241 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.832246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.832249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.832584 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:46.832695 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.833664 (+ 969us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.833886 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11663,"thread_start_us":16652,"threads_started":1}
I20260925 11:09:46.836380 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.795637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38197 (local address 127.8.181.3:53653)
0925 11:09:46.811295 (+ 15658us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:46.811380 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:46.814956 (+ 3576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:46.816072 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:46.816084 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:46.816691 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:46.817530 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.817557 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.819215 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.819221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.821449 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.821463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.822758 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.822764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.823062 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.824381 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:46.824415 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:46.827116 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:46.830793 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:46.830806 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:46.830828 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:46.831452 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:46.831857 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:46.831861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:46.831865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:46.832117 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:46.832796 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:46.832804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:46.833207 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 11:09:46.834450 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0925 11:09:46.835017 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15220,"thread_start_us":14970,"threads_started":1}
I20260925 11:09:46.838142 22289 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38197
I20260925 11:09:46.838503 22289 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:46.839306 22289 heartbeater.cc:511] Master 127.8.181.62:38197 requested a full tablet report, sending...
I20260925 11:09:46.841014 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.799242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.8.181.62:38197)
0925 11:09:46.811989 (+ 12747us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.811993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.812006 (+ 13us) server_negotiation.cc:407] Connection header received
0925 11:09:46.812062 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.812066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.812148 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.812288 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:46.814485 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.815570 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.817604 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.818393 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.821976 (+ 3583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.823533 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:46.823541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:46.823548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:46.823587 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:46.831428 (+ 7841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.837177 (+ 5749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.837182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.837186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.837383 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.838061 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.838065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.838069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.838655 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:46.838911 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.840348 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.840595 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12643,"thread_start_us":7363,"threads_started":1}
I20260925 11:09:46.841187 21814 ts_manager.cc:194] Registered new tserver with Master: 2cc84c780a9044ce84319857afc00d13 (127.8.181.3:44369)
I20260925 11:09:46.842394 21814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:53653
I20260925 11:09:46.845705 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:09:46.868444 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.849322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.8.181.62:38197)
0925 11:09:46.849498 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.849504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.850027 (+ 523us) server_negotiation.cc:407] Connection header received
0925 11:09:46.850189 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.850194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.850279 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.850384 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:46.852439 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.853453 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.855427 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.856092 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.856195 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.860119 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:46.860126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:46.860130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:46.860175 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:46.863948 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.864990 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.864996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.864999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.865153 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:46.865728 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:46.865732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:46.865735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:46.866091 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:46.866388 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.867726 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.867954 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 11:09:46.875319 21814 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:09:46.887214 21814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 25 11:09:46 dist-test-slave-6042 krb5kdc[20803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for HTTP/127.8.181.20@KRBTEST.COM
I20260925 11:09:46.925200 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.914553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:44379 (local address 127.0.0.1:40000)
0925 11:09:46.915343 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:46.915397 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:46.915585 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:46.916763 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:46.916792 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:46.916830 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:46.917344 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.917358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.919544 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.919549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.921981 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.922000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.924182 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.924190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.924417 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.924480 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 11:09:46.924603 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 11:09:46.924823 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":583,"threads_started":1}
I20260925 11:09:46.926931 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.914669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.8.181.2:44379)
0925 11:09:46.916123 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.916132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.916160 (+ 28us) server_negotiation.cc:407] Connection header received
0925 11:09:46.916257 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.916264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.916512 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.917551 (+ 1039us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:46.917782 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.919360 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.922218 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.923913 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.926120 (+ 2207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.926340 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.926416 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.926623 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":1192,"threads_started":1}
I20260925 11:09:46.931608 22090 tablet_service.cc:1467] Processing CreateTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 (DEFAULT_TABLE table=test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:46.933419 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73c36d1645448aea3e2b23e0d3fba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:46.954020 22308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.919925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44369 (local address 127.0.0.1:55946)
0925 11:09:46.944666 (+ 24741us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:46.944698 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:46.944872 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:46.946645 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:46.946650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:46.946677 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:46.946988 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.947014 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.948954 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.948960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.951027 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.951044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.953230 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.953236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.953408 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.953441 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:09:46.953517 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 11:09:46.953656 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24591,"thread_start_us":25055,"threads_started":1}
I20260925 11:09:46.954842 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.920469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.8.181.3:44369)
0925 11:09:46.927017 (+ 6548us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.927030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.944822 (+ 17792us) server_negotiation.cc:407] Connection header received
0925 11:09:46.945036 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.945043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.945261 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.945456 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:46.947249 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.948769 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.951219 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.952697 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.952796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.953041 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.954323 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.954523 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6344,"thread_start_us":6194,"threads_started":1}
I20260925 11:09:46.958834 22224 tablet_service.cc:1467] Processing CreateTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 (DEFAULT_TABLE table=test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:46.959892 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.916962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41379 (local address 127.0.0.1:50312)
0925 11:09:46.925809 (+ 8847us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:46.925830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:46.925948 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:46.944951 (+ 19003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:46.944956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:46.944988 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:46.945226 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.945239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.955151 (+ 9912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.955158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.956911 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:46.956928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.959035 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.959042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:46.959244 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.959287 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 11:09:46.959388 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 11:09:46.959547 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8727,"thread_start_us":28659,"threads_started":1}
I20260925 11:09:46.960413 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73c36d1645448aea3e2b23e0d3fba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:46.967334 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:46.930849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.8.181.1:41379)
0925 11:09:46.944228 (+ 13379us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:46.944239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:46.944287 (+ 48us) server_negotiation.cc:407] Connection header received
0925 11:09:46.944407 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:46.944416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:46.944716 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:46.944960 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:46.946709 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.955024 (+ 8315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.957261 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:46.958834 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:46.958985 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:46.959232 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:46.966684 (+ 7452us) server_negotiation.cc:299] Negotiation successful
0925 11:09:46.966920 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13173,"thread_start_us":14492,"threads_started":1}
I20260925 11:09:46.971856 21946 tablet_service.cc:1467] Processing CreateTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 (DEFAULT_TABLE table=test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:46.973680 21946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73c36d1645448aea3e2b23e0d3fba88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:47.014659 22312 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Bootstrap starting.
I20260925 11:09:47.015954 22311 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Bootstrap starting.
I20260925 11:09:47.018834 22312 tablet_bootstrap.cc:654] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:47.020342 22312 log.cc:824] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:47.021116 22311 tablet_bootstrap.cc:654] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:47.022346 22311 log.cc:824] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:47.027788 22311 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: No bootstrap required, opened a new log
I20260925 11:09:47.028121 22311 ts_tablet_manager.cc:1397] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260925 11:09:47.031595 22313 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Bootstrap starting.
I20260925 11:09:47.038801 22313 tablet_bootstrap.cc:654] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:47.040207 22313 log.cc:824] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:47.050554 22311 raft_consensus.cc:348] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.051095 22311 raft_consensus.cc:374] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:47.051194 22311 raft_consensus.cc:729] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc84c780a9044ce84319857afc00d13, State: Initialized, Role: FOLLOWER
I20260925 11:09:47.051885 22311 consensus_queue.cc:260] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.052574 22312 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: No bootstrap required, opened a new log
I20260925 11:09:47.052953 22312 ts_tablet_manager.cc:1397] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.016s
I20260925 11:09:47.055305 22313 tablet_bootstrap.cc:492] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: No bootstrap required, opened a new log
I20260925 11:09:47.055624 22313 ts_tablet_manager.cc:1397] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20260925 11:09:47.065797 22312 raft_consensus.cc:348] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.066417 22312 raft_consensus.cc:374] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:47.066601 22312 raft_consensus.cc:729] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431a79cc568340e3af2e993a2c3ad3ba, State: Initialized, Role: FOLLOWER
I20260925 11:09:47.067474 22312 consensus_queue.cc:260] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.085220 22155 heartbeater.cc:503] Master 127.8.181.62:38197 was elected leader, sending a full tablet report...
I20260925 11:09:47.085978 22312 ts_tablet_manager.cc:1428] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Time spent starting tablet: real 0.033s user 0.012s sys 0.017s
I20260925 11:09:47.086301 22313 raft_consensus.cc:348] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.086959 22313 raft_consensus.cc:374] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:47.087088 22313 raft_consensus.cc:729] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955e4e298e3e4a6a8177cdd46eb3672f, State: Initialized, Role: FOLLOWER
I20260925 11:09:47.088271 22289 heartbeater.cc:503] Master 127.8.181.62:38197 was elected leader, sending a full tablet report...
I20260925 11:09:47.088964 22311 ts_tablet_manager.cc:1428] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Time spent starting tablet: real 0.061s user 0.022s sys 0.025s
I20260925 11:09:47.089694 22313 consensus_queue.cc:260] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.096800 22011 heartbeater.cc:503] Master 127.8.181.62:38197 was elected leader, sending a full tablet report...
I20260925 11:09:47.097873 22313 ts_tablet_manager.cc:1428] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Time spent starting tablet: real 0.042s user 0.020s sys 0.000s
W20260925 11:09:47.115226 22156 tablet.cc:2369] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:09:47.135565 22012 tablet.cc:2369] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:09:47.213095 22318 raft_consensus.cc:482] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:09:47.213457 22318 raft_consensus.cc:504] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.215754 22318 leader_election.cc:288] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 955e4e298e3e4a6a8177cdd46eb3672f (127.8.181.1:41379), 2cc84c780a9044ce84319857afc00d13 (127.8.181.3:44369)
I20260925 11:09:47.235721 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.216740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:34743 (local address 127.8.181.1:41379)
0925 11:09:47.216949 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.216955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.226163 (+ 9208us) server_negotiation.cc:407] Connection header received
0925 11:09:47.226400 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.226406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.226545 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.226659 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:47.227499 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.228592 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.233116 (+ 4524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.234296 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.234456 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.234598 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.235260 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.235424 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 11:09:47.235891 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.217211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:58927 (local address 127.8.181.3:44369)
0925 11:09:47.217402 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.217407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.227121 (+ 9714us) server_negotiation.cc:407] Connection header received
0925 11:09:47.227393 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.227400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.227479 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.227540 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:47.228306 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.229338 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.233153 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.234456 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.234610 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.234743 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.235468 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.235635 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:09:47.236490 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.216119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41379 (local address 127.8.181.2:34743)
0925 11:09:47.226008 (+ 9889us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:47.226035 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:47.226239 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:47.226846 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:47.226852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:47.226882 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:47.227303 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:47.227316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.228768 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.228773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:47.232947 (+ 4174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:47.232962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.234783 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.234789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:47.235045 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.235085 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 11:09:47.235930 (+ 845us) client_negotiation.cc:240] Negotiation successful
0925 11:09:47.236080 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9715,"mutex_wait_us":2154,"thread_start_us":13557,"threads_started":1}
I20260925 11:09:47.237510 21966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73c36d1645448aea3e2b23e0d3fba88" candidate_uuid: "431a79cc568340e3af2e993a2c3ad3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" is_pre_election: true
I20260925 11:09:47.238119 21966 raft_consensus.cc:2454] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431a79cc568340e3af2e993a2c3ad3ba in term 0.
I20260925 11:09:47.239054 22045 leader_election.cc:302] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431a79cc568340e3af2e993a2c3ad3ba, 955e4e298e3e4a6a8177cdd46eb3672f; no voters:
I20260925 11:09:47.239524 22318 raft_consensus.cc:2781] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:09:47.239655 22318 raft_consensus.cc:482] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:09:47.239737 22318 raft_consensus.cc:3037] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:47.242702 22318 raft_consensus.cc:504] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.243053 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.216419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44369 (local address 127.8.181.2:58927)
0925 11:09:47.227002 (+ 10583us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:47.227025 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:47.227224 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:47.227745 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:47.227749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:47.227774 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:47.228127 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:47.228137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.229555 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.229561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:47.231710 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:47.231724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.235086 (+ 3362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.235093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:47.235273 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.235304 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 11:09:47.236539 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0925 11:09:47.242611 (+ 6072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10445,"mutex_wait_us":1,"thread_start_us":9930,"threads_started":1}
I20260925 11:09:47.244233 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73c36d1645448aea3e2b23e0d3fba88" candidate_uuid: "431a79cc568340e3af2e993a2c3ad3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc84c780a9044ce84319857afc00d13" is_pre_election: true
I20260925 11:09:47.244750 22244 raft_consensus.cc:2454] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431a79cc568340e3af2e993a2c3ad3ba in term 0.
I20260925 11:09:47.245519 21966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73c36d1645448aea3e2b23e0d3fba88" candidate_uuid: "431a79cc568340e3af2e993a2c3ad3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "955e4e298e3e4a6a8177cdd46eb3672f"
I20260925 11:09:47.245779 22318 leader_election.cc:288] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [CANDIDATE]: Term 1 election: Requested vote from peers 955e4e298e3e4a6a8177cdd46eb3672f (127.8.181.1:41379), 2cc84c780a9044ce84319857afc00d13 (127.8.181.3:44369)
I20260925 11:09:47.245777 21966 raft_consensus.cc:3037] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:47.247073 22244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73c36d1645448aea3e2b23e0d3fba88" candidate_uuid: "431a79cc568340e3af2e993a2c3ad3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc84c780a9044ce84319857afc00d13"
I20260925 11:09:47.247294 22244 raft_consensus.cc:3037] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:47.248320 21966 raft_consensus.cc:2454] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431a79cc568340e3af2e993a2c3ad3ba in term 1.
I20260925 11:09:47.248872 22045 leader_election.cc:302] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431a79cc568340e3af2e993a2c3ad3ba, 955e4e298e3e4a6a8177cdd46eb3672f; no voters:
I20260925 11:09:47.249224 22318 raft_consensus.cc:2781] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:09:47.250049 22318 raft_consensus.cc:686] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 1 LEADER]: Becoming Leader. State: Replica: 431a79cc568340e3af2e993a2c3ad3ba, State: Running, Role: LEADER
I20260925 11:09:47.250754 22318 consensus_queue.cc:237] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } }
I20260925 11:09:47.251547 22244 raft_consensus.cc:2454] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431a79cc568340e3af2e993a2c3ad3ba in term 1.
I20260925 11:09:47.255529 21813 catalog_manager.cc:5495] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 431a79cc568340e3af2e993a2c3ad3ba (127.8.181.2). New cstate: current_term: 1 leader_uuid: "431a79cc568340e3af2e993a2c3ad3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431a79cc568340e3af2e993a2c3ad3ba" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 } health_report { overall_health: UNKNOWN } } }
W20260925 11:09:47.300100 22290 tablet.cc:2369] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:09:47.336534 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.330216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.8.181.2:44379)
0925 11:09:47.330447 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.330455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.330515 (+ 60us) server_negotiation.cc:407] Connection header received
0925 11:09:47.330649 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.330671 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.330791 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.330953 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:09:47.331366 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.332444 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.333803 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.334443 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.334595 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.335188 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:09:47.335902 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:09:47.336054 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.336132 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.336220 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:09:47.345077 22244 raft_consensus.cc:1261] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 1 FOLLOWER]: Refusing update from remote peer 431a79cc568340e3af2e993a2c3ad3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:09:47.346145 22318 consensus_queue.cc:1035] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc84c780a9044ce84319857afc00d13" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:09:47.346585 21966 raft_consensus.cc:1261] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 1 FOLLOWER]: Refusing update from remote peer 431a79cc568340e3af2e993a2c3ad3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:09:47.347442 22328 consensus_queue.cc:1035] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "955e4e298e3e4a6a8177cdd46eb3672f" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:09:47.360512 22334 mvcc.cc:205] Tried to move back new op lower bound from 7333210469755215872 to 7333210469383651328. Current Snapshot: MvccSnapshot[applied={T|T < 7333210469755215872}]
I20260925 11:09:47.361364 22332 mvcc.cc:205] Tried to move back new op lower bound from 7333210469755215872 to 7333210469383651328. Current Snapshot: MvccSnapshot[applied={T|T < 7333210469755215872}]
I20260925 11:09:47.394972 21813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:09:47.395272 21813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:09:47.399668 21813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9122c816fdf444aa5ca336ce5dfc83a: Sending DeleteTablet for 3 replicas of tablet d73c36d1645448aea3e2b23e0d3fba88
I20260925 11:09:47.400966 21946 tablet_service.cc:1514] Processing DeleteTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:09:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50312
I20260925 11:09:47.400966 22090 tablet_service.cc:1514] Processing DeleteTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:09:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260925 11:09:47.401777 22224 tablet_service.cc:1514] Processing DeleteTablet for tablet d73c36d1645448aea3e2b23e0d3fba88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:09:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:55946
I20260925 11:09:47.409251 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21851
I20260925 11:09:47.415688 22343 tablet_replica.cc:330] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: stopping tablet replica
I20260925 11:09:47.416333 22343 raft_consensus.cc:2229] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:09:47.416981 22343 raft_consensus.cc:2258] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:09:47.419869 22341 tablet_replica.cc:330] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: stopping tablet replica
I20260925 11:09:47.420324 22341 raft_consensus.cc:2229] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:09:47.420823 22341 raft_consensus.cc:2258] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:09:47.421545 22343 ts_tablet_manager.cc:1905] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:09:47.426645 22342 tablet_replica.cc:330] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: stopping tablet replica
I20260925 11:09:47.427101 22342 raft_consensus.cc:2229] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:09:47.427765 22342 raft_consensus.cc:2258] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:09:47.428313 22341 ts_tablet_manager.cc:1905] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:09:47.436641 22341 ts_tablet_manager.cc:1918] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:09:47.436995 22341 log.cc:1196] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/d73c36d1645448aea3e2b23e0d3fba88
I20260925 11:09:47.437696 22342 ts_tablet_manager.cc:1905] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:09:47.437794 22341 ts_tablet_manager.cc:1939] T d73c36d1645448aea3e2b23e0d3fba88 P 955e4e298e3e4a6a8177cdd46eb3672f: Deleting consensus metadata
I20260925 11:09:47.440294 21800 catalog_manager.cc:4841] TS 955e4e298e3e4a6a8177cdd46eb3672f (127.8.181.1:41379): tablet d73c36d1645448aea3e2b23e0d3fba88 (table test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]) successfully deleted
I20260925 11:09:47.441111 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.410320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.8.181.1:41379)
0925 11:09:47.410585 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.410595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.423029 (+ 12434us) server_negotiation.cc:407] Connection header received
0925 11:09:47.423261 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.423267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.423372 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.423504 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:47.425632 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.426788 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.428586 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.429346 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.429491 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.432793 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:47.432818 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:47.432837 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:47.432882 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:47.437003 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.437951 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.437962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.437966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.438135 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.438603 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.438608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.438612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.439138 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:47.439505 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.439763 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.440706 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:09:47.441807 22006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:47.442641 22343 ts_tablet_manager.cc:1918] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:09:47.442847 22343 log.cc:1196] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/d73c36d1645448aea3e2b23e0d3fba88
I20260925 11:09:47.443418 22343 ts_tablet_manager.cc:1939] T d73c36d1645448aea3e2b23e0d3fba88 P 2cc84c780a9044ce84319857afc00d13: Deleting consensus metadata
I20260925 11:09:47.445307 21799 catalog_manager.cc:4841] TS 2cc84c780a9044ce84319857afc00d13 (127.8.181.3:44369): tablet d73c36d1645448aea3e2b23e0d3fba88 (table test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]) successfully deleted
I20260925 11:09:47.453456 22342 ts_tablet_manager.cc:1918] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:09:47.453691 22342 log.cc:1196] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/d73c36d1645448aea3e2b23e0d3fba88
I20260925 11:09:47.454298 22342 ts_tablet_manager.cc:1939] T d73c36d1645448aea3e2b23e0d3fba88 P 431a79cc568340e3af2e993a2c3ad3ba: Deleting consensus metadata
I20260925 11:09:47.458483 21801 catalog_manager.cc:4841] TS 431a79cc568340e3af2e993a2c3ad3ba (127.8.181.2:44379): tablet d73c36d1645448aea3e2b23e0d3fba88 (table test-table [id=ef3f1f1f6fbb45869fe2c979e1a6fab6]) successfully deleted
I20260925 11:09:47.607824 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21851
I20260925 11:09:47.635481 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22015
I20260925 11:09:47.655737 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.637087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.8.181.2:44379)
0925 11:09:47.637277 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.637283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.637302 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:09:47.637371 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.637376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.637458 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.637553 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:47.639710 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.640989 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.643217 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.643976 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.644124 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.647620 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:47.647641 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:47.647662 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:47.647719 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:47.652043 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.653058 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.653067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.653071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.653290 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.653753 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.653758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.653760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.654227 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:47.654591 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.654898 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.655127 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:09:47.656441 22150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:47.773980 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22015
I20260925 11:09:47.793792 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22159
I20260925 11:09:47.809096 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:47.794590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.8.181.3:44369)
0925 11:09:47.795125 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:47.795129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:47.795146 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:09:47.795194 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:47.795198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:47.795254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:47.795349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:47.797059 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.797836 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.799634 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:47.800061 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:47.800187 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:47.803366 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:47.803377 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:47.803391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:47.803424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:47.806273 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.807208 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.807216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.807218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.807335 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:47.807694 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:47.807697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:47.807699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:47.808045 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:47.808271 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:47.808549 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 11:09:47.808719 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":310,"threads_started":1}
I20260925 11:09:47.809816 22284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:47.898916 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22159
I20260925 11:09:47.915387 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 21781
I20260925 11:09:47.916617 21844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:48.035317 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 21781
I20260925 11:09:48.061185 8916 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 11:09:48.397276 8916 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 11:09:48.397543 8916 mini_ranger.cc:66] Stopping Ranger...
I20260925 11:09:48.743132 8916 mini_ranger.cc:68] Stopped Ranger
2026-09-25 11:09:48.743 UTC [20833] LOG: received smart shutdown request
2026-09-25 11:09:48.747 UTC [20833] LOG: background worker "logical replication launcher" (PID 20842) exited with exit code 1
2026-09-25 11:09:48.747 UTC [20837] LOG: shutting down
2026-09-25 11:09:48.954 UTC [20833] LOG: database system is shut down
2026-09-25T11:09:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56456 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:09:49 dist-test-slave-6042 krb5kdc[22358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:09:49 dist-test-slave-6042 krb5kdc[22358](info): set up 2 sockets
Sep 25 11:09:49 dist-test-slave-6042 krb5kdc[22358](info): commencing operation
krb5kdc: starting...
W20260925 11:09:50.191318 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:09:50 dist-test-slave-6042 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:09:50.438263 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40619
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:50.770714 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:50.771024 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:50.771224 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:50.778637 22371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:50.778795 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:50.778888 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:50.778973 22371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:50.779036 22371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:50.789233 22371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:50.790675 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:50.792013 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:50.804491 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:50.804459 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:50.804437 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:50.805423 22371 server_base.cc:1034] running on GCE node
I20260925 11:09:50.806223 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:50.807780 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:50.809019 22371 hybrid_clock.cc:648] HybridClock initialized: now 1790334590808966 us; error 59 us; skew 500 ppm
I20260925 11:09:50.809365 22371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:50 dist-test-slave-6042 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:50.817513 22371 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:09:50.819085 22371 webserver.cc:460] Webserver started at http://127.8.181.62:40845/ using document root <none> and password file <none>
I20260925 11:09:50.819762 22371 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:50.819865 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:50.820106 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:50.823257 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "2c526c56908b405ea251e05010e25ef2"
format_stamp: "Formatted at 2026-09-25 11:09:50 on dist-test-slave-6042"
server_key: "91a09e472f9cb1b632f492550d5abd82"
server_key_iv: "f6cff0131d83e19f2156676b2dc7c525"
server_key_version: "encryptionkey@0"
I20260925 11:09:50.824079 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "2c526c56908b405ea251e05010e25ef2"
format_stamp: "Formatted at 2026-09-25 11:09:50 on dist-test-slave-6042"
server_key: "91a09e472f9cb1b632f492550d5abd82"
server_key_iv: "f6cff0131d83e19f2156676b2dc7c525"
server_key_version: "encryptionkey@0"
I20260925 11:09:50.829253 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:09:50.832739 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:50.834167 22371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 11:09:50.834554 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "2c526c56908b405ea251e05010e25ef2"
format_stamp: "Formatted at 2026-09-25 11:09:50 on dist-test-slave-6042"
server_key: "91a09e472f9cb1b632f492550d5abd82"
server_key_iv: "f6cff0131d83e19f2156676b2dc7c525"
server_key_version: "encryptionkey@0"
I20260925 11:09:50.834751 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:50.856426 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:50.861025 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:50.861327 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:50.880308 22371 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:40619
I20260925 11:09:50.880343 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:40619 every 8 connection(s)
I20260925 11:09:50.881683 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:09:50.885457 22438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:50.891921 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22371
I20260925 11:09:50.892170 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:09:50.893002 8916 external_mini_cluster.cc:1428] Setting key bb8ab46d05b69b9c18deb87f277097a8
I20260925 11:09:50.892267 22438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:09:50 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.61@KRBTEST.COM" created.
I20260925 11:09:50.920279 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.896617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40619 (local address 127.0.0.1:59278)
0925 11:09:50.897663 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:50.897719 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:50.900763 (+ 3044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:50.901992 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:50.901997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:50.902705 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:50.903204 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:50.903229 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:50.904940 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:50.904948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:50.907133 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:50.907153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:50.908216 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:50.908226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:50.908461 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:50.909270 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:50.909319 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:50.912181 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:50.915835 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:50.915847 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:50.915860 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:50.916260 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:50.916671 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:50.916679 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:50.916684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:50.916857 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:50.917675 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:50.917683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:50.917947 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 11:09:50.918202 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 11:09:50.918843 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":143,"thread_start_us":500,"threads_started":1}
I20260925 11:09:50.920295 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.896617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.8.181.62:40619)
0925 11:09:50.898871 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:50.898877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:50.898925 (+ 48us) server_negotiation.cc:407] Connection header received
0925 11:09:50.901345 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:50.901355 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:50.901724 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:50.901976 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:50.903422 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:50.904775 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:50.907311 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:50.907942 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:50.908089 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:50.912468 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:50.912477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:50.912491 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:50.912525 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:50.915667 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:50.916410 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:50.916416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:50.916419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:50.916558 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:50.916980 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:50.916989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:50.916991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:50.917528 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:50.917794 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:50.918336 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 11:09:50.918738 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":2038,"spinlock_wait_cycles":83328,"thread_start_us":1787,"threads_started":1}
I20260925 11:09:50.923651 22438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:09:50.924988 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.924494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45362)
0925 11:09:50.924672 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:50.924866 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 11:09:50.925279 22388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
I20260925 11:09:50.926937 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.926529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45376)
0925 11:09:50.926678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:50.926749 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 11:09:50.927188 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:42075: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
I20260925 11:09:50.950968 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:42075
--webserver_interface=127.8.181.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:09:50.976584 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } attempt: 1
I20260925 11:09:50.977886 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.977334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45390)
0925 11:09:50.977555 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:50.977667 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 11:09:50.979020 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:50.978581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45404)
0925 11:09:50.978808 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:50.978878 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 11:09:50.979262 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:42075: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
I20260925 11:09:51.047535 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } attempt: 2
I20260925 11:09:51.048784 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.048359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45416)
0925 11:09:51.048552 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.048640 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 11:09:51.049772 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.049420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45420)
0925 11:09:51.049561 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.049651 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 11:09:51.049960 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:42075: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
I20260925 11:09:51.141207 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } attempt: 3
I20260925 11:09:51.142634 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.142099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45428)
0925 11:09:51.142308 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.142418 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 11:09:51.143630 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.143275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45438)
0925 11:09:51.143418 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.143497 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 11:09:51.143805 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:42075: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
W20260925 11:09:51.285601 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:51.285950 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:51.286118 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 11:09:51.287025 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } attempt: 4
I20260925 11:09:51.288169 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.287663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45452)
0925 11:09:51.287964 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.288049 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 11:09:51.289175 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.288796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45466)
0925 11:09:51.288971 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.289054 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 11:09:51.289357 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:42075: Network error: Client connection negotiation failed: client connection to 127.8.181.61:42075: connect: Connection refused (error 111)
W20260925 11:09:51.293757 22445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:51.293902 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:51.294016 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:51.294112 22445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:51.294201 22445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:51.304278 22445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:42075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.181.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:51.305667 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:51.307173 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:51.319685 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:51.319733 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:51.321376 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:51.322099 22445 server_base.cc:1034] running on GCE node
I20260925 11:09:51.323071 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:51.324517 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:51.325778 22445 hybrid_clock.cc:648] HybridClock initialized: now 1790334591325702 us; error 88 us; skew 500 ppm
I20260925 11:09:51.326157 22445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:51.329550 22445 init.cc:377] Logged in from keytab as kudu/127.8.181.61@KRBTEST.COM (short username kudu)
I20260925 11:09:51.331316 22445 webserver.cc:460] Webserver started at http://127.8.181.61:36489/ using document root <none> and password file <none>
I20260925 11:09:51.332185 22445 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:51.332332 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:51.332640 22445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:51.335798 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data/instance:
uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "819eead70da26c76b0dd7b3c8dbe737c"
server_key_iv: "e21a6865623b4efdcdb986c68b1f5e54"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.336618 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal/instance:
uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "819eead70da26c76b0dd7b3c8dbe737c"
server_key_iv: "e21a6865623b4efdcdb986c68b1f5e54"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.342197 22445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:09:51.345927 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:51.347494 22445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 11:09:51.347677 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "819eead70da26c76b0dd7b3c8dbe737c"
server_key_iv: "e21a6865623b4efdcdb986c68b1f5e54"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.347890 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:51.361809 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:51.366909 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:51.367273 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:51.387722 22445 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.61:42075
I20260925 11:09:51.387741 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.61:42075 every 8 connection(s)
I20260925 11:09:51.389245 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
I20260925 11:09:51.393071 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:51.394542 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22445
I20260925 11:09:51.394742 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-1/wal/instance
I20260925 11:09:51.395350 8916 external_mini_cluster.cc:1428] Setting key abb4c0fd2788465c9af75116a7945956
I20260925 11:09:51.403029 22512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.181.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.60@KRBTEST.COM" created.
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:51.428646 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.406509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.8.181.62:40619)
0925 11:09:51.406733 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.406741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.408188 (+ 1447us) server_negotiation.cc:407] Connection header received
0925 11:09:51.410424 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.410432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.410565 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.410703 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.412457 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.413611 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.415536 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.416229 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.416386 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.422739 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.422745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.422748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.422785 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.425516 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.426312 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.426317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.426320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.426441 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.426927 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.426932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.426935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.427217 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.427385 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.428094 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.428319 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 11:09:51.429649 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.407039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40619 (local address 127.0.0.1:59284)
0925 11:09:51.407987 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.408042 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.410129 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.411056 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.411067 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.411648 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.412264 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.412282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.413892 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.413897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.415379 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.415392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.416460 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.416466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.416745 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.417605 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.417632 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:51.422419 (+ 4787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:51.425677 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.425687 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.425706 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.426167 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.426604 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.426609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.426613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.426801 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.427352 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:51.427362 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:51.427731 (+ 369us) client_negotiation.cc:769] Sending connection context
0925 11:09:51.427997 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 11:09:51.428521 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":457,"threads_started":1}
I20260925 11:09:51.431734 22512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:09:51.450363 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.432625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45470)
0925 11:09:51.432885 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.432909 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.433048 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.434071 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.434080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.434702 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.434940 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.434957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.436382 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.436391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.438252 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.438271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.439501 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.439513 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.439748 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.440439 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.440472 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:51.443290 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:51.447692 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.447696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.447699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.448184 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.448598 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.448601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.448604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.448747 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.449540 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:51.449543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:51.449659 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 11:09:51.449856 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 11:09:51.450085 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 11:09:51.450791 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.432845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.8.181.61:42075)
0925 11:09:51.433594 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.433600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.433625 (+ 25us) server_negotiation.cc:407] Connection header received
0925 11:09:51.433672 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.433677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.433844 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.433966 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.435119 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.436231 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.438413 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.439327 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.439426 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.443559 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.443581 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.443607 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.443659 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.447535 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.448351 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.448364 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.448366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.448488 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.448941 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.448946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.448948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.449390 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.449566 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.450040 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.450244 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":456,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
I20260925 11:09:51.453452 22512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:09:51.454535 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:35519
--webserver_interface=127.8.181.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:09:51.454666 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.454066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58842)
0925 11:09:51.454287 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.454435 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 11:09:51.454957 22462 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.456671 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.456200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58848)
0925 11:09:51.456420 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.456528 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 11:09:51.456936 22512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.506307 22512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 1
I20260925 11:09:51.507654 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.507189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58864)
0925 11:09:51.507391 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.507500 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 11:09:51.508728 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.508357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58872)
0925 11:09:51.508523 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.508616 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 11:09:51.508965 22512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.577246 22512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 2
I20260925 11:09:51.578610 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.578003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58878)
0925 11:09:51.578251 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.578368 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 11:09:51.579782 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.579376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58888)
0925 11:09:51.579542 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.579629 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 11:09:51.580029 22512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.588624 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } attempt: 5
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:09:51.604216 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.589492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.8.181.61:42075)
0925 11:09:51.589668 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.589680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.589706 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:09:51.589969 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.589974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.590063 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.590162 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.591355 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.592173 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.593994 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.594458 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.594623 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.598542 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.598553 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.598556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.598591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.601388 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.602214 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.602219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.602221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.602376 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.602821 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.602826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.602830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.603118 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.603232 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.603717 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.603932 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 11:09:51.604234 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.589373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45482)
0925 11:09:51.589608 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.589635 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.589794 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.590234 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.590241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.590861 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.591155 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.591168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.592350 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.592359 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.593827 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.593846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.594688 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.594694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.594866 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.595516 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.595539 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:51.598242 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:51.601553 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.601559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.601564 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.602078 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.602511 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.602516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.602520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.602700 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.603241 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:51.603246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:51.603420 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 11:09:51.603579 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 11:09:51.603889 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 11:09:51.605608 22438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:09:51.606858 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.606265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58904)
0925 11:09:51.606513 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.606629 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 11:09:51.607991 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.607571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58918)
0925 11:09:51.607761 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.607867 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 11:09:51.608240 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.659502 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 1
I20260925 11:09:51.660864 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.660309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58932)
0925 11:09:51.660538 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.660662 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 11:09:51.662027 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.661637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58940)
0925 11:09:51.661798 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.661891 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 11:09:51.662261 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.671289 22512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 3
I20260925 11:09:51.672566 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.671978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58942)
0925 11:09:51.672248 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.672369 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 11:09:51.673897 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.673397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58958)
0925 11:09:51.673645 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.673749 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260925 11:09:51.674131 22512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
I20260925 11:09:51.730598 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 2
I20260925 11:09:51.731907 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.731300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58960)
0925 11:09:51.731585 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.731703 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 11:09:51.733134 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.732735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58970)
0925 11:09:51.732895 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.732999 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 11:09:51.733407 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
W20260925 11:09:51.796861 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:51.797147 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:51.797338 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:51.804852 22519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:51.804955 22519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:51.805065 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:51.805145 22519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:51.805210 22519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:51.815915 22519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40619,127.8.181.61:42075,127.8.181.60:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.181.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:51.817682 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:51.818277 22512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 4
I20260925 11:09:51.819608 22519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 11:09:51.821157 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.820441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58982)
0925 11:09:51.820816 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.820935 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260925 11:09:51.823771 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.822201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:58994)
0925 11:09:51.822339 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.822442 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 11:09:51.824028 22512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
W20260925 11:09:51.833523 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:51.833871 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:51.835021 22519 server_base.cc:1034] running on GCE node
W20260925 11:09:51.835832 22527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:51.837050 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:51.839156 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:51.839641 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 3
I20260925 11:09:51.840420 22519 hybrid_clock.cc:648] HybridClock initialized: now 1790334591840346 us; error 61 us; skew 500 ppm
I20260925 11:09:51.840785 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.840291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:59010)
0925 11:09:51.840513 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.840611 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 11:09:51.840916 22519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:09:51.841874 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.841457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:59024)
0925 11:09:51.841631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.841759 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 11:09:51.842442 22438 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:35519: Network error: Client connection negotiation failed: client connection to 127.8.181.60:35519: connect: Connection refused (error 111)
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:51.845238 22519 init.cc:377] Logged in from keytab as kudu/127.8.181.60@KRBTEST.COM (short username kudu)
I20260925 11:09:51.847223 22519 webserver.cc:460] Webserver started at http://127.8.181.60:37487/ using document root <none> and password file <none>
I20260925 11:09:51.847972 22519 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:51.848119 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:51.848429 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:51.851763 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data/instance:
uuid: "611bcb92da364c6999135cace07315d9"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "ec61de596cdc0fcecc1e475ca49658e3"
server_key_iv: "9a82a5125a665cfb4494c48dbe11d628"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.852635 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal/instance:
uuid: "611bcb92da364c6999135cace07315d9"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "ec61de596cdc0fcecc1e475ca49658e3"
server_key_iv: "9a82a5125a665cfb4494c48dbe11d628"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.858579 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260925 11:09:51.862664 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:51.864320 22519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:09:51.864519 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "611bcb92da364c6999135cace07315d9"
format_stamp: "Formatted at 2026-09-25 11:09:51 on dist-test-slave-6042"
server_key: "ec61de596cdc0fcecc1e475ca49658e3"
server_key_iv: "9a82a5125a665cfb4494c48dbe11d628"
server_key_version: "encryptionkey@0"
I20260925 11:09:51.864727 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:51.895995 22519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:51.900799 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:51.901099 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:51.922169 22519 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.60:35519
I20260925 11:09:51.922187 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.60:35519 every 8 connection(s)
I20260925 11:09:51.923678 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
I20260925 11:09:51.929145 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:51.929693 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22519
I20260925 11:09:51.929983 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790334376327139-8916-0/minicluster-data/master-2/wal/instance
I20260925 11:09:51.930546 8916 external_mini_cluster.cc:1428] Setting key c64bf47346f625e4e6346d768ebc72c9
I20260925 11:09:51.941382 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:51.967579 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.946776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.8.181.62:40619)
0925 11:09:51.947381 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.947390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.948179 (+ 789us) server_negotiation.cc:407] Connection header received
0925 11:09:51.948988 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.948994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.949071 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.949199 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.950606 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.951479 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.954129 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.954987 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.955106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.959969 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.959979 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.959987 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.960039 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.964290 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.965185 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.965194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.965200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.965395 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.965900 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.965906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.965909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.966188 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.966358 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.966901 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.967149 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":283,"threads_started":1}
I20260925 11:09:51.968181 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.932885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.8.181.60:35519)
0925 11:09:51.943666 (+ 10781us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.943680 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.943732 (+ 52us) server_negotiation.cc:407] Connection header received
0925 11:09:51.945669 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.945694 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.946202 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.948348 (+ 2146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.948986 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.951435 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.953098 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.953883 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.954084 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.958917 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.958932 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.958955 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.959002 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.962324 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.963143 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.963153 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.963167 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.963328 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.963745 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.963757 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.963761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.964338 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.964665 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.965929 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.966558 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":10478,"thread_start_us":10201,"threads_started":1}
I20260925 11:09:51.968222 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.947273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40619 (local address 127.0.0.1:59290)
0925 11:09:51.948000 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.948065 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.948830 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.949289 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.949297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.949991 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.950376 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.950390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.951732 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.951740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.953937 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.953957 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.955485 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.955495 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.955738 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.956523 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.956560 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:51.959711 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:51.964448 (+ 4737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.964457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.964463 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.965034 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.965541 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.965546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.965549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.965745 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.966337 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:51.966345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:51.966570 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 11:09:51.966764 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 11:09:51.967155 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":74,"thread_start_us":273,"threads_started":1}
I20260925 11:09:51.970772 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:51.988983 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.971680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:42075 (local address 127.0.0.1:45484)
0925 11:09:51.971972 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.971993 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.972178 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.972799 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.972805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.973343 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.973582 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.973595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.974841 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.974847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.975973 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.975988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.977353 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.977362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.977590 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.978358 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.978385 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:51.981574 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:51.985915 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.985924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.985930 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.986687 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.987182 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:51.987187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:51.987191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:51.987354 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:51.987899 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:51.987904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:51.988095 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 11:09:51.988252 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 11:09:51.988552 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260925 11:09:51.990088 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.971795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.8.181.61:42075)
0925 11:09:51.971980 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.971985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.972066 (+ 81us) server_negotiation.cc:407] Connection header received
0925 11:09:51.972330 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.972339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.972444 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.972532 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.973776 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.974672 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.976148 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.977175 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.977328 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.981852 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.981865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.981869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.981905 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.985719 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.986854 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.986862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.986866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.987073 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.987477 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.987482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.987486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.987776 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.987933 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.989396 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.989647 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:09:51.991158 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.972233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.8.181.62:40619)
0925 11:09:51.972390 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.972396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.972413 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:09:51.972500 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.972505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.972589 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.972695 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.974010 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.975324 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.976839 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.977297 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.977419 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.983132 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:51.983139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:51.983144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:51.983196 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:51.986977 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.988090 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.988095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.988099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.988273 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:51.988784 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:51.988789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:51.988792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:51.989169 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:51.989460 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:51.990066 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 11:09:51.990721 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 11:09:51.991169 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:09:51 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
Sep 25 11:09:52 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:09:52.009199 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.992369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:59036)
0925 11:09:51.992592 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:51.992614 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:51.992735 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:51.993300 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:51.993305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:51.993787 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:51.993924 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.993932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.994985 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.994993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.996083 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:51.996099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.997265 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.997272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:51.997475 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:51.998197 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:51.998222 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:52.000772 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:52.005758 (+ 4986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.005765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.005769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.006609 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.007088 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.007093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.007096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.007271 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.008010 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:52.008015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:52.008203 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 11:09:52.008389 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 11:09:52.008754 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 11:09:52.009605 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.992702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.8.181.60:35519)
0925 11:09:51.992866 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.992872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.992890 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:09:51.992939 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.992944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.993017 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.993125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.994063 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.994860 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.996244 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.997094 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.997243 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.001026 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.001036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.001044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.001092 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.005578 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.006767 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.006776 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.006782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.006950 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.007407 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.007415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.007420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.007883 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.008056 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.008518 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.009043 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":60}
I20260925 11:09:52.011402 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:51.993160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.8.181.61:42075)
0925 11:09:51.993315 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:51.993321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:51.993335 (+ 14us) server_negotiation.cc:407] Connection header received
0925 11:09:51.993384 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:51.993389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:51.993459 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:51.993561 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:51.995642 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.996640 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.998221 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:51.999081 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:51.999248 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.004425 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.004431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.004434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.004473 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.007777 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.008751 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.008756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.008760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.008928 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.009413 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.009420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.009426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.009820 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.010103 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.010749 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.011002 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 11:09:52.015787 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: Bootstrap starting.
I20260925 11:09:52.017217 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:09:52.019393 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:52.019711 22438 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 4
I20260925 11:09:52.021016 22586 log.cc:824] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:09:52 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334590, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:09:52.036600 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: No bootstrap required, opened a new log
I20260925 11:09:52.042614 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.020347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:59046)
0925 11:09:52.020748 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:52.020770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:52.020893 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:52.021900 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:52.021908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:52.022674 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:52.022998 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:52.023015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.025233 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.025242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:52.027332 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:52.027356 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.028631 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.028641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:52.028915 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.029737 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:52.029773 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:52.033028 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:52.038165 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.038175 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.038182 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.038998 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.039735 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.039744 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.039750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.039978 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.041049 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:52.041057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:52.041325 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 11:09:52.041498 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 11:09:52.041956 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260925 11:09:52.044938 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.021201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.8.181.60:35519)
0925 11:09:52.021437 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:52.021443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:52.021463 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:09:52.021532 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:52.021537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:52.021643 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:52.023226 (+ 1583us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:52.023432 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.024966 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.027597 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.028379 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.029378 (+ 999us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.033295 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.033304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.033311 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.033362 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.037903 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.039326 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.039335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.039342 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.039565 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.040170 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.040178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.040188 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.040844 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.043714 (+ 2870us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.043933 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.044335 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:09:52.051678 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.052445 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:52.052634 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611bcb92da364c6999135cace07315d9, State: Initialized, Role: FOLLOWER
I20260925 11:09:52.053670 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.054822 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: Bootstrap starting.
I20260925 11:09:52.061653 22438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:52.063547 22438 log.cc:824] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:52.063896 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:52.061661 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.065912 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.088762 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: No bootstrap required, opened a new log
I20260925 11:09:52.097002 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.067873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.8.181.60:35519)
0925 11:09:52.068032 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:52.068040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:52.074967 (+ 6927us) server_negotiation.cc:407] Connection header received
0925 11:09:52.075417 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:52.075423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:52.075556 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:52.075628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:52.078308 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.080053 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.082525 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.083876 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.084010 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.088111 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.088119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.088124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.088171 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.092853 (+ 4682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.094124 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.094131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.094136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.094306 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.094903 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.094909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.094913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.095334 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.095598 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.096281 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.096576 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 11:09:52.105345 22438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.106011 22438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:52.106201 22438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c526c56908b405ea251e05010e25ef2, State: Initialized, Role: FOLLOWER
I20260925 11:09:52.107010 22438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.111814 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.036861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.8.181.61:42075)
0925 11:09:52.037128 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:52.037138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:52.067417 (+ 30279us) server_negotiation.cc:407] Connection header received
0925 11:09:52.067860 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:52.067870 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:52.067988 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:52.068091 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:52.078911 (+ 10820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.080316 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.084080 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.085070 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.085211 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.100255 (+ 15044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.100266 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.100274 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.100332 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.105326 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.107065 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.107072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.107076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.107296 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.107984 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.107993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.107999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.108589 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.108954 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.110757 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.111156 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 11:09:52.114710 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.046130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.8.181.62:40619)
0925 11:09:52.046400 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:52.046409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:52.067794 (+ 21385us) server_negotiation.cc:407] Connection header received
0925 11:09:52.068281 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:52.068288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:52.068415 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:52.068513 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:52.074393 (+ 5880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.075536 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.080428 (+ 4892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.081103 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.081236 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.098281 (+ 17045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.098291 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.098296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.098350 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.103221 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.104324 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.104330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.104335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.104522 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.105036 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.105041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.105044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.105502 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.113512 (+ 8010us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.113730 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.114055 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 11:09:52.116129 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.117576 22438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:52.118045 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.123291 22512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } attempt: 5
Sep 25 11:09:52 dist-test-slave-6042 krb5kdc[22358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334591, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
W20260925 11:09:52.153678 22540 tablet.cc:2369] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:09:52.154510 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.123988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:35519 (local address 127.0.0.1:59068)
0925 11:09:52.124290 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:52.124310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:52.124427 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:52.125045 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:52.125053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:52.125828 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:52.126120 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:52.126136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.128351 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.128360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:52.130593 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:52.130617 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.134649 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.134660 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:52.134976 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.135927 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:52.135961 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:52.142565 (+ 6604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:52.150106 (+ 7541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.150114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.150120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.151012 (+ 892us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.151717 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:52.151726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:52.151732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:52.151941 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:52.153013 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:52.153020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:52.153253 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 11:09:52.153453 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 11:09:52.153882 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260925 11:09:52.155642 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:52.124184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.8.181.60:35519)
0925 11:09:52.124352 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:52.124358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:52.124376 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:09:52.124613 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:52.124622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:52.124726 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:52.124831 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:52.126362 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.128047 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.133187 (+ 5140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:52.134155 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:52.134330 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:52.142910 (+ 8580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:52.142922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:52.142930 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:52.142989 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:52.149936 (+ 6947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.151270 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.151280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.151286 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.151553 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:52.152104 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:52.152112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:52.152118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:52.152737 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:52.152942 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:52.154932 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0925 11:09:52.155217 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 11:09:52.173197 22618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:09:52.173445 22618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:09:52.175019 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: Bootstrap starting.
I20260925 11:09:52.187345 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:52.189145 22512 log.cc:824] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:52.205118 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: No bootstrap required, opened a new log
I20260925 11:09:52.165575 22519 master_runner.cc:186] Error getting master registration for 127.8.181.62:40619: OK, instance_id { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" instance_seqno: 1790334590875082 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:09:52.207865 22519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:09:52.220631 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.221500 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:52.221737 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1415fc5c78b44d2834bbc0d9f96c5e6, State: Initialized, Role: FOLLOWER
I20260925 11:09:52.222750 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.232570 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.233497 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.234597 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 11:09:52.238528 22627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:09:52.238730 22627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:09:52.236655 22371 master_runner.cc:186] Error getting master registration for 127.8.181.61:42075: OK, instance_id { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" instance_seqno: 1790334591382035 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:09:52.248492 22371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:09:52.276607 22445 master_runner.cc:263] No leader master found from master d1415fc5c78b44d2834bbc0d9f96c5e6
I20260925 11:09:52.276779 22445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:09:52.285230 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 11:09:52.285420 22639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:09:52.285568 22639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:09:52.285549 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.287298 22598 leader_election.cc:288] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c526c56908b405ea251e05010e25ef2 (127.8.181.62:40619), d1415fc5c78b44d2834bbc0d9f96c5e6 (127.8.181.61:42075)
I20260925 11:09:52.288072 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "611bcb92da364c6999135cace07315d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c526c56908b405ea251e05010e25ef2" is_pre_election: true
I20260925 11:09:52.288673 22413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611bcb92da364c6999135cace07315d9 in term 0.
I20260925 11:09:52.288586 22487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "611bcb92da364c6999135cace07315d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" is_pre_election: true
I20260925 11:09:52.289180 22487 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 611bcb92da364c6999135cace07315d9 in term 0.
I20260925 11:09:52.289499 22538 leader_election.cc:302] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c526c56908b405ea251e05010e25ef2, 611bcb92da364c6999135cace07315d9; no voters:
I20260925 11:09:52.289909 22598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:09:52.290071 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:09:52.290179 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:52.293890 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.295365 22413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "611bcb92da364c6999135cace07315d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c526c56908b405ea251e05010e25ef2"
I20260925 11:09:52.295661 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:52.299422 22413 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611bcb92da364c6999135cace07315d9 in term 1.
I20260925 11:09:52.299921 22538 leader_election.cc:302] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c526c56908b405ea251e05010e25ef2, 611bcb92da364c6999135cace07315d9; no voters:
I20260925 11:09:52.302901 22487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "611bcb92da364c6999135cace07315d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6"
I20260925 11:09:52.303167 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:52.304236 22598 leader_election.cc:288] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [CANDIDATE]: Term 1 election: Requested vote from peers 2c526c56908b405ea251e05010e25ef2 (127.8.181.62:40619), d1415fc5c78b44d2834bbc0d9f96c5e6 (127.8.181.61:42075)
I20260925 11:09:52.304478 22598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:09:52.304618 22598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [term 1 LEADER]: Becoming Leader. State: Replica: 611bcb92da364c6999135cace07315d9, State: Running, Role: LEADER
I20260925 11:09:52.305207 22598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } }
I20260925 11:09:52.306762 22487 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 611bcb92da364c6999135cace07315d9 in term 1.
I20260925 11:09:52.315215 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611bcb92da364c6999135cace07315d9. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.315632 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:52.332058 22643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:09:52.334743 22643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:09:52.358717 22487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [term 1 FOLLOWER]: Refusing update from remote peer 611bcb92da364c6999135cace07315d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:09:52.360000 22641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 11:09:52.364420 22392 tablet.cc:2369] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:09:52.370178 22466 tablet.cc:2369] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:09:52.371306 22413 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [term 1 FOLLOWER]: Refusing update from remote peer 611bcb92da364c6999135cace07315d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:09:52.372396 22641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:09:52.395934 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611bcb92da364c6999135cace07315d9. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.396284 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.402885 22641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.403203 22641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:52.405481 22644 mvcc.cc:205] Tried to move back new op lower bound from 7333210490287706112 to 7333210490095931392. Current Snapshot: MvccSnapshot[applied={T|T < 7333210490287706112}]
I20260925 11:09:52.409169 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611bcb92da364c6999135cace07315d9. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.409548 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.411218 22643 catalog_manager.cc:1347] Generated new cluster ID: cff9de593f0641a5bf100d464c31c4bc
I20260925 11:09:52.411324 22643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:09:52.411260 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.411580 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:52.413883 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.414143 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.414870 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "611bcb92da364c6999135cace07315d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c526c56908b405ea251e05010e25ef2" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40619 } } peers { permanent_uuid: "d1415fc5c78b44d2834bbc0d9f96c5e6" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 42075 } } peers { permanent_uuid: "611bcb92da364c6999135cace07315d9" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 35519 } } }
I20260925 11:09:52.415133 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:09:52.428469 22646 mvcc.cc:205] Tried to move back new op lower bound from 7333210490287706112 to 7333210490095931392. Current Snapshot: MvccSnapshot[applied={T|T < 7333210490287706112 or (T in {7333210490287706112})}]
I20260925 11:09:52.445834 22643 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:09:52.448020 22643 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:09:52.468092 22643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611bcb92da364c6999135cace07315d9: Generated new TSK 0
I20260925 11:09:52.469532 22643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:09:53.241103 22627 catalog_manager.cc:1259] Loaded cluster ID: cff9de593f0641a5bf100d464c31c4bc
I20260925 11:09:53.241250 22627 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: loading cluster ID for follower catalog manager: success
I20260925 11:09:53.243856 22627 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: acquiring CA information for follower catalog manager: success
I20260925 11:09:53.245734 22627 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c526c56908b405ea251e05010e25ef2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:09:53.287628 22639 catalog_manager.cc:1259] Loaded cluster ID: cff9de593f0641a5bf100d464c31c4bc
I20260925 11:09:53.287814 22639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: loading cluster ID for follower catalog manager: success
I20260925 11:09:53.290381 22639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: acquiring CA information for follower catalog manager: success
I20260925 11:09:53.292241 22639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1415fc5c78b44d2834bbc0d9f96c5e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:09:53.617475 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22371
I20260925 11:09:53.618840 22433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:53.729146 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22371
I20260925 11:09:53.752197 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22445
I20260925 11:09:53.753343 22507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:53.860896 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22445
I20260925 11:09:53.882261 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22519
I20260925 11:09:53.883374 22581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:53.993613 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22519
2026-09-25T11:09:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4925 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:09:54 dist-test-slave-6042 krb5kdc[22662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:09:54 dist-test-slave-6042 krb5kdc[22662](info): set up 2 sockets
Sep 25 11:09:54 dist-test-slave-6042 krb5kdc[22662](info): commencing operation
krb5kdc: starting...
W20260925 11:09:55.155373 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:09:55 dist-test-slave-6042 krb5kdc[22662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:09:55.411583 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34187
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:55.750916 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:55.751223 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:55.751403 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:55.758914 22675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:55.759034 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:55.759130 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:55.759238 22675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:55.759320 22675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:55.769310 22675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:55.770700 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:55.772012 22675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:55.784003 22680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:55.784130 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:55.784162 22681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:55.785382 22675 server_base.cc:1034] running on GCE node
I20260925 11:09:55.786172 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:55.787640 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:55.788872 22675 hybrid_clock.cc:648] HybridClock initialized: now 1790334595788798 us; error 53 us; skew 500 ppm
I20260925 11:09:55.789178 22675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:55 dist-test-slave-6042 krb5kdc[22662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334595, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:55.792693 22675 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:09:55.794420 22675 webserver.cc:460] Webserver started at http://127.8.181.62:38341/ using document root <none> and password file <none>
I20260925 11:09:55.795168 22675 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:55.795317 22675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:55.795625 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:55.798861 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "b9f6fa1cda514c4697e05678eb45c0d8"
format_stamp: "Formatted at 2026-09-25 11:09:55 on dist-test-slave-6042"
server_key: "bc62b9728769914c9304f1cd5d9a970c"
server_key_iv: "f5c7497bd107a2aba8c24cbfb261bfa3"
server_key_version: "encryptionkey@0"
I20260925 11:09:55.799618 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "b9f6fa1cda514c4697e05678eb45c0d8"
format_stamp: "Formatted at 2026-09-25 11:09:55 on dist-test-slave-6042"
server_key: "bc62b9728769914c9304f1cd5d9a970c"
server_key_iv: "f5c7497bd107a2aba8c24cbfb261bfa3"
server_key_version: "encryptionkey@0"
I20260925 11:09:55.804800 22675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:09:55.808218 22690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:55.809544 22675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:09:55.809734 22675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "b9f6fa1cda514c4697e05678eb45c0d8"
format_stamp: "Formatted at 2026-09-25 11:09:55 on dist-test-slave-6042"
server_key: "bc62b9728769914c9304f1cd5d9a970c"
server_key_iv: "f5c7497bd107a2aba8c24cbfb261bfa3"
server_key_version: "encryptionkey@0"
I20260925 11:09:55.809942 22675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:55.839027 22675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:55.843618 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:55.843922 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:55.863344 22675 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34187
I20260925 11:09:55.863387 22741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34187 every 8 connection(s)
I20260925 11:09:55.864779 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:09:55.866358 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22675
I20260925 11:09:55.866616 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:09:55.867273 8916 external_mini_cluster.cc:1428] Setting key 96489358ad43bb66b92edbe777b0bd26
I20260925 11:09:55.869323 22742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 11:09:55 dist-test-slave-6042 krb5kdc[22662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:55.883581 22742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8: Bootstrap starting.
I20260925 11:09:55.887024 22742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:55.888087 22742 log.cc:824] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:55.891352 22742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8: No bootstrap required, opened a new log
I20260925 11:09:55.891780 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:55.869246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.8.181.62:34187)
0925 11:09:55.870117 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:55.870130 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:55.870170 (+ 40us) server_negotiation.cc:407] Connection header received
0925 11:09:55.871873 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:55.871894 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:55.872285 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:55.872624 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:55.874611 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:55.876016 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:55.877895 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:55.878639 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:55.878771 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:55.883899 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:55.883908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:55.883924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:55.883965 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:55.887394 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:55.888183 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:55.888191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:55.888205 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:55.888425 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:55.888824 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:55.888830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:55.888835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:55.889400 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:55.889708 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:55.890190 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 11:09:55.890657 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":396,"threads_started":1}
I20260925 11:09:55.896044 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:09:55.900382 22742 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } }
I20260925 11:09:55.900799 22742 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:55.900907 22742 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f6fa1cda514c4697e05678eb45c0d8, State: Initialized, Role: FOLLOWER
I20260925 11:09:55.901535 22742 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } }
I20260925 11:09:55.901782 22742 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:09:55.901932 22742 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:09:55.902122 22742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:55.904398 22742 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } }
I20260925 11:09:55.904922 22742 leader_election.cc:302] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f6fa1cda514c4697e05678eb45c0d8; no voters:
I20260925 11:09:55.905534 22742 leader_election.cc:288] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:09:55.905690 22747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:09:55.906423 22747 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [term 1 LEADER]: Becoming Leader. State: Replica: b9f6fa1cda514c4697e05678eb45c0d8, State: Running, Role: LEADER
I20260925 11:09:55.907029 22747 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } }
I20260925 11:09:55.907789 22742 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:55.910310 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f6fa1cda514c4697e05678eb45c0d8. Latest consensus state: current_term: 1 leader_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } } }
I20260925 11:09:55.910807 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:55.911345 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f6fa1cda514c4697e05678eb45c0d8" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34187 } } }
I20260925 11:09:55.911760 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:55.912627 22754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:09:55.922216 22754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:09:55.931945 22754 catalog_manager.cc:1347] Generated new cluster ID: 679c7d0d10744e7c84ed31aab3b9e8db
I20260925 11:09:55.932109 22754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:09:55.942554 22754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:09:55.943936 22754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:09:55.951364 22754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f6fa1cda514c4697e05678eb45c0d8: Generated new TSK 0
I20260925 11:09:55.952509 22754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:09:56.009438 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22675
I20260925 11:09:56.010900 22737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:09:56.110263 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22675
2026-09-25T11:09:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2103 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:09:56 dist-test-slave-6042 krb5kdc[22770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:09:56 dist-test-slave-6042 krb5kdc[22770](info): set up 2 sockets
Sep 25 11:09:56 dist-test-slave-6042 krb5kdc[22770](info): commencing operation
krb5kdc: starting...
W20260925 11:09:57.271476 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:09:57 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:09:57.530524 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:57.863863 22783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:57.864164 22783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:57.864253 22783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:57.871587 22783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:09:57.871708 22783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:57.871757 22783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:57.871816 22783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:09:57.871861 22783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:09:57.882019 22783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:57.883488 22783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:57.884778 22783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:57.897149 22788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:57.897189 22791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:57.897122 22789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:57.898021 22783 server_base.cc:1034] running on GCE node
I20260925 11:09:57.898972 22783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:57.900424 22783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:57.901661 22783 hybrid_clock.cc:648] HybridClock initialized: now 1790334597901611 us; error 32 us; skew 500 ppm
I20260925 11:09:57.902062 22783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:57 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:57.905483 22783 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:09:57.907178 22783 webserver.cc:460] Webserver started at http://127.8.181.62:44841/ using document root <none> and password file <none>
I20260925 11:09:57.907915 22783 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:57.908053 22783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:57.908380 22783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:57.911598 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "a4de74eae4a8417cbfed00bc2e21ba61"
format_stamp: "Formatted at 2026-09-25 11:09:57 on dist-test-slave-6042"
server_key: "18097937c1c3b85937d021604719dffd"
server_key_iv: "897696643e0821a9e15840b9cffbb5b9"
server_key_version: "encryptionkey@0"
I20260925 11:09:57.912431 22783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "a4de74eae4a8417cbfed00bc2e21ba61"
format_stamp: "Formatted at 2026-09-25 11:09:57 on dist-test-slave-6042"
server_key: "18097937c1c3b85937d021604719dffd"
server_key_iv: "897696643e0821a9e15840b9cffbb5b9"
server_key_version: "encryptionkey@0"
I20260925 11:09:57.918016 22783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:09:57.921896 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:57.923666 22783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:09:57.923974 22783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a4de74eae4a8417cbfed00bc2e21ba61"
format_stamp: "Formatted at 2026-09-25 11:09:57 on dist-test-slave-6042"
server_key: "18097937c1c3b85937d021604719dffd"
server_key_iv: "897696643e0821a9e15840b9cffbb5b9"
server_key_version: "encryptionkey@0"
I20260925 11:09:57.924227 22783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:57.948542 22783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:57.953873 22783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:57.954198 22783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:57.975072 22783 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:44335
I20260925 11:09:57.975118 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:44335 every 8 connection(s)
I20260925 11:09:57.976680 22783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:09:57.980964 22850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:57.984697 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22783
I20260925 11:09:57.984964 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:09:57.985632 8916 external_mini_cluster.cc:1428] Setting key 3223531debe992731dfa0b4a6d33f5d7
I20260925 11:09:57.995985 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap starting.
I20260925 11:09:58.000193 22850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:58.001375 22850 log.cc:824] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:09:58 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:58.005084 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: No bootstrap required, opened a new log
I20260925 11:09:58.013557 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:57.988150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.8.181.62:44335)
0925 11:09:57.989004 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:57.989020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:57.989076 (+ 56us) server_negotiation.cc:407] Connection header received
0925 11:09:57.991249 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:57.991301 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:57.992009 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:57.992583 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:57.994151 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:57.996286 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:57.997900 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:57.999026 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:57.999252 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:58.003774 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:58.003787 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:58.003807 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:58.003854 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:58.008364 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:58.009261 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:58.009271 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:58.009281 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:58.009512 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:58.009976 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:58.009981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:58.009985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:58.010815 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:58.011159 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:58.011551 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 11:09:58.012059 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":340,"threads_started":1}
I20260925 11:09:58.016240 22850 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:09:58.016719 22850 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:58.016849 22850 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Initialized, Role: FOLLOWER
I20260925 11:09:58.017443 22850 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:09:58.017709 22850 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:09:58.017870 22850 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:09:58.018046 22850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:09:58.020368 22850 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:09:58.020920 22850 leader_election.cc:302] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4de74eae4a8417cbfed00bc2e21ba61; no voters:
I20260925 11:09:58.021538 22850 leader_election.cc:288] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:09:58.021763 22855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:09:58.022979 22855 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 LEADER]: Becoming Leader. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Running, Role: LEADER
I20260925 11:09:58.023746 22855 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:09:58.023963 22850 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:09:58.027503 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:09:58.027926 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:58.027051 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4de74eae4a8417cbfed00bc2e21ba61. Latest consensus state: current_term: 1 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:09:58.028277 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:09:58.030303 22863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:09:58.036499 22863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:09:58.046799 22863 catalog_manager.cc:1347] Generated new cluster ID: 40156d3ba3a54071af075d6462fa4341
I20260925 11:09:58.046955 22863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:09:58.062398 22863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:09:58.064332 22863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:09:58.077755 22863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Generated new TSK 0
I20260925 11:09:58.078824 22863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:09:58.147378 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:58.500582 22876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:58.501020 22876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:58.501226 22876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:58.508812 22876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:58.508927 22876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:58.509123 22876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:09:58.519383 22876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:58.520762 22876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:58.522104 22876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:58.539243 22884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:58.539331 22882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:58.539444 22881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:58.540493 22876 server_base.cc:1034] running on GCE node
I20260925 11:09:58.541170 22876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:58.542155 22876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:58.543393 22876 hybrid_clock.cc:648] HybridClock initialized: now 1790334598543320 us; error 73 us; skew 500 ppm
I20260925 11:09:58.543737 22876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:58 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:58.547353 22876 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:09:58.549163 22876 webserver.cc:460] Webserver started at http://127.8.181.1:36299/ using document root <none> and password file <none>
I20260925 11:09:58.549935 22876 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:58.550087 22876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:58.550398 22876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:58.553570 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "a95f155cbfca486b92797d11bf397ca5"
format_stamp: "Formatted at 2026-09-25 11:09:58 on dist-test-slave-6042"
server_key: "13a001651154365df0658df18cf559e5"
server_key_iv: "a5e3d8d5b51179851323b5a93be4bfe0"
server_key_version: "encryptionkey@0"
I20260925 11:09:58.554437 22876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "a95f155cbfca486b92797d11bf397ca5"
format_stamp: "Formatted at 2026-09-25 11:09:58 on dist-test-slave-6042"
server_key: "13a001651154365df0658df18cf559e5"
server_key_iv: "a5e3d8d5b51179851323b5a93be4bfe0"
server_key_version: "encryptionkey@0"
I20260925 11:09:58.560280 22876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:09:58.564255 22891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:58.565894 22876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:09:58.566099 22876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "a95f155cbfca486b92797d11bf397ca5"
format_stamp: "Formatted at 2026-09-25 11:09:58 on dist-test-slave-6042"
server_key: "13a001651154365df0658df18cf559e5"
server_key_iv: "a5e3d8d5b51179851323b5a93be4bfe0"
server_key_version: "encryptionkey@0"
I20260925 11:09:58.566335 22876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:58.588711 22876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:58.593441 22876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:58.593766 22876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:58.594858 22876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:58.596253 22876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:58.596330 22876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:58.596503 22876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:58.596583 22876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:58.635998 22876 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34751
I20260925 11:09:58.636094 23003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34751 every 8 connection(s)
I20260925 11:09:58.637398 22876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:09:58.639569 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 22876
I20260925 11:09:58.639803 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:09:58.640693 8916 external_mini_cluster.cc:1428] Setting key 398a2b4f3b7e1c77da4fa7dba6df73cf
Sep 25 11:09:58 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:58.663509 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:58.640281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:37743 (local address 127.8.181.62:44335)
0925 11:09:58.640774 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:58.640782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:58.642122 (+ 1340us) server_negotiation.cc:407] Connection header received
0925 11:09:58.644395 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:58.644404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:58.644503 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:58.644632 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:58.646258 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:58.647512 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:58.650455 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:58.650996 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:58.651162 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:58.655762 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:58.655770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:58.655775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:58.655821 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:58.659166 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:58.660396 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:58.660401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:58.660404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:58.660568 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:58.661197 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:58.661201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:58.661203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:58.661504 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:58.661645 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:58.662804 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0925 11:09:58.663028 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":267,"threads_started":1}
I20260925 11:09:58.665012 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:58.641009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:37743)
0925 11:09:58.641942 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:58.641992 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:58.644073 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:58.644858 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:58.644867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:58.645429 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:58.646020 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:58.646037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:58.647711 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:58.647718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:58.650197 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:58.650222 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:58.651314 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:58.651324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:58.651611 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:58.652459 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:58.652489 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:58.655422 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:58.659395 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:58.659410 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:58.659434 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:58.660207 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:58.660733 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:58.660741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:58.660747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:58.661055 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:58.661677 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:58.661689 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:58.662257 (+ 568us) client_negotiation.cc:769] Sending connection context
0925 11:09:58.662662 (+ 405us) client_negotiation.cc:240] Negotiation successful
0925 11:09:58.663337 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":402,"threads_started":1}
I20260925 11:09:58.666891 23004 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
I20260925 11:09:58.667238 23004 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:58.668089 23004 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
I20260925 11:09:58.670706 22815 ts_manager.cc:194] Registered new tserver with Master: a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751)
I20260925 11:09:58.673031 22815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:37743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:09:58.706521 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:59.055496 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:59.055859 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:59.056041 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:59.063905 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:59.064088 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:59.064289 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:09:59.074862 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:59.076270 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:59.077600 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:59.091995 23015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:59.092027 23016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:59.092060 23018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:59.092907 23010 server_base.cc:1034] running on GCE node
I20260925 11:09:59.093609 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:59.094648 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:59.095921 23010 hybrid_clock.cc:648] HybridClock initialized: now 1790334599095885 us; error 39 us; skew 500 ppm
I20260925 11:09:59.096290 23010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:59 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:59.099762 23010 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:09:59.101382 23010 webserver.cc:460] Webserver started at http://127.8.181.2:38023/ using document root <none> and password file <none>
I20260925 11:09:59.102064 23010 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:59.102164 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:59.102418 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:59.105492 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "c7a6ddfe52111dd57cf63e94e496346e"
server_key_iv: "11567058c1d0e7b55e2bb049089d975a"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.106269 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "c7a6ddfe52111dd57cf63e94e496346e"
server_key_iv: "11567058c1d0e7b55e2bb049089d975a"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.112084 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 11:09:59.116024 23025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.117834 23010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260925 11:09:59.118144 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "c7a6ddfe52111dd57cf63e94e496346e"
server_key_iv: "11567058c1d0e7b55e2bb049089d975a"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.118420 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:59.142046 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:59.146934 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:59.147281 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:59.148350 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:59.149781 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:59.149861 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.150010 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:59.150064 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.185595 23010 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:34383
I20260925 11:09:59.185693 23137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:34383 every 8 connection(s)
I20260925 11:09:59.186966 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:09:59.191713 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23010
I20260925 11:09:59.191936 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:09:59.192369 8916 external_mini_cluster.cc:1428] Setting key ed8cf7d4783b37ff56dc14becebc1e44
Sep 25 11:09:59 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:59.210402 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.189937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35685 (local address 127.8.181.62:44335)
0925 11:09:59.190531 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.190538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.191263 (+ 725us) server_negotiation.cc:407] Connection header received
0925 11:09:59.193544 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.193548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.193619 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.193710 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:59.195601 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.196892 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.198793 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.199637 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.199790 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.203989 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:59.203994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:59.203997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:59.204032 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:59.206819 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.207915 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.207920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.207922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.208048 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.208588 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.208591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.208594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.208857 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:59.209046 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.209945 (+ 899us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.210131 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":313,"threads_started":1}
I20260925 11:09:59.212006 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.190260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:35685)
0925 11:09:59.191080 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:59.191123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:59.193192 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:59.194039 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:59.194051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:59.194758 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:59.195403 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.195427 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.197084 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.197088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.198627 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.198640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.199904 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.199910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.200217 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.201002 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:59.201026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:59.203661 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:59.206975 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:59.206988 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:59.207010 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:59.207733 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:59.208168 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:59.208177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:59.208183 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:59.208457 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:59.209038 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:59.209050 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:59.209534 (+ 484us) client_negotiation.cc:769] Sending connection context
0925 11:09:59.209851 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 11:09:59.210492 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":371,"threads_started":1}
I20260925 11:09:59.213686 23138 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
I20260925 11:09:59.214047 23138 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:59.214823 23138 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
I20260925 11:09:59.216357 22815 ts_manager.cc:194] Registered new tserver with Master: c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:09:59.217672 22815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:35685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:09:59.254546 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:09:59.604013 23144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:09:59.604316 23144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:09:59.604411 23144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:09:59.611905 23144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:09:59.612037 23144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:09:59.612151 23144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:09:59.622396 23144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:09:59.624035 23144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:09:59.625439 23144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:09:59.639864 23149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:59.639962 23150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:09:59.639889 23152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:09:59.641547 23144 server_base.cc:1034] running on GCE node
I20260925 11:09:59.642225 23144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:09:59.643316 23144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:09:59.644608 23144 hybrid_clock.cc:648] HybridClock initialized: now 1790334599644584 us; error 73 us; skew 500 ppm
I20260925 11:09:59.644994 23144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:09:59 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:09:59.648914 23144 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:09:59.650648 23144 webserver.cc:460] Webserver started at http://127.8.181.3:34071/ using document root <none> and password file <none>
I20260925 11:09:59.651517 23144 fs_manager.cc:359] Metadata directory not provided
I20260925 11:09:59.651652 23144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:09:59.651985 23144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:09:59.655151 23144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "8d1aeafaf9e812a63e1234c1409379fd"
server_key_iv: "f2802d9e928450e200b7363d2ea9ad26"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.655937 23144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "8d1aeafaf9e812a63e1234c1409379fd"
server_key_iv: "f2802d9e928450e200b7363d2ea9ad26"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.661417 23144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:09:59.665077 23159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.666649 23144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:09:59.666901 23144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "8d1aeafaf9e812a63e1234c1409379fd"
server_key_iv: "f2802d9e928450e200b7363d2ea9ad26"
server_key_version: "encryptionkey@0"
I20260925 11:09:59.667128 23144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:09:59.677407 23004 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:09:59.688035 23144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:09:59.693017 23144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:09:59.693320 23144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:09:59.694360 23144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:09:59.695910 23144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:09:59.695991 23144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.696089 23144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:09:59.696138 23144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:09:59.734759 23144 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45243
I20260925 11:09:59.734791 23271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45243 every 8 connection(s)
I20260925 11:09:59.736159 23144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:09:59.742883 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23144
I20260925 11:09:59.743075 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:09:59.743525 8916 external_mini_cluster.cc:1428] Setting key a730c0d0d3c2388c14381eeb6ab953d7
Sep 25 11:09:59 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334599, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:09:59.758709 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.738753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:53373 (local address 127.8.181.62:44335)
0925 11:09:59.739230 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.739236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.740281 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 11:09:59.742330 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.742334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.742513 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.742647 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:59.744422 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.745563 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.747703 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.748129 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.748228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.752206 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:59.752212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:59.752215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:59.752254 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:59.755154 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.755980 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.755985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.755987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.756107 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.756541 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.756545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.756547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.756845 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:59.756967 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.758077 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.758298 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":199,"threads_started":1}
I20260925 11:09:59.760051 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.739305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:53373)
0925 11:09:59.740105 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:59.740144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:59.742005 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:59.742936 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:59.742946 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:59.743650 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:09:59.744261 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.744279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.745856 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.745862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.747569 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.747583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.748352 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.748357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.748561 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.749289 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:09:59.749320 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:09:59.751925 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:09:59.755327 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:59.755336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:59.755353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:59.755850 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:59.756225 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:09:59.756228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:09:59.756231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:09:59.756425 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:09:59.756974 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:09:59.756981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:09:59.757598 (+ 617us) client_negotiation.cc:769] Sending connection context
0925 11:09:59.757879 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 11:09:59.758568 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":271,"threads_started":1}
I20260925 11:09:59.761544 23272 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
I20260925 11:09:59.761845 23272 heartbeater.cc:464] Registering TS with master...
I20260925 11:09:59.762672 23272 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
I20260925 11:09:59.764180 22815 ts_manager.cc:194] Registered new tserver with Master: 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243)
I20260925 11:09:59.764977 22815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:53373
I20260925 11:09:59.772421 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:09:59.790318 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.777094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.8.181.62:44335)
0925 11:09:59.777451 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.777457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.777643 (+ 186us) server_negotiation.cc:407] Connection header received
0925 11:09:59.777848 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.777852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.777920 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.778007 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:09:59.779552 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.780337 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.781979 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.782401 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.782607 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.785100 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:09:59.785105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:09:59.785108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:09:59.785143 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:09:59.787853 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.788491 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.788495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.788498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.788618 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:09:59.789001 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:09:59.789004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:09:59.789006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:09:59.789342 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:09:59.789533 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.789832 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.790002 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260925 11:09:59.794680 22815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:09:59.801509 22815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:09:59.838244 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.817280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.8.181.2:34383)
0925 11:09:59.829855 (+ 12575us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.829868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.829901 (+ 33us) server_negotiation.cc:407] Connection header received
0925 11:09:59.830001 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.830008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.830228 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.830431 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:59.831421 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.832766 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.835485 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837020 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.837158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.837378 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.837719 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.837890 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12355,"thread_start_us":12165,"threads_started":1}
I20260925 11:09:59.839053 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.814980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.8.181.3:45243)
0925 11:09:59.817004 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.825234 (+ 8230us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.828648 (+ 3414us) server_negotiation.cc:407] Connection header received
0925 11:09:59.829261 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.829275 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.829640 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.829873 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:59.831253 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.833595 (+ 2342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.836143 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837049 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.837207 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.837447 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.838259 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.838711 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1651,"threads_started":1}
I20260925 11:09:59.839265 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.814404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34751 (local address 127.0.0.1:50640)
0925 11:09:59.815244 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:59.815324 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:59.815555 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:59.830959 (+ 15404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:59.830964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:59.830988 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:59.831471 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.831487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.834024 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.834030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.836095 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.836109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837966 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.837972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.838149 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.838207 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 11:09:59.838270 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 11:09:59.838414 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":40,"thread_start_us":389,"threads_started":1}
I20260925 11:09:59.839344 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.814926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.8.181.1:34751)
0925 11:09:59.830256 (+ 15330us) server_negotiation.cc:206] Beginning negotiation
0925 11:09:59.830272 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 11:09:59.830293 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:09:59.830431 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:09:59.830440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:09:59.830661 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:09:59.830958 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:09:59.831663 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.833902 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.836284 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837725 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.838579 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.838798 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 11:09:59.838874 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 11:09:59.839023 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15166,"thread_start_us":15007,"threads_started":1}
I20260925 11:09:59.839728 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.815506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45243 (local address 127.0.0.1:53472)
0925 11:09:59.828473 (+ 12967us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:59.828504 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:59.828997 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:59.830244 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:59.830254 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:59.830286 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:59.830986 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.830998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.833748 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.833762 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.835982 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.835993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837548 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.837555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.837738 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.837781 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 11:09:59.837858 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 11:09:59.838020 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12764,"mutex_wait_us":241,"thread_start_us":13189,"threads_started":1}
I20260925 11:09:59.840054 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:09:59.816694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:34383 (local address 127.0.0.1:33290)
0925 11:09:59.828852 (+ 12158us) negotiation.cc:107] Waiting for socket to connect
0925 11:09:59.828877 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:09:59.829033 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:09:59.830494 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:09:59.830499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:09:59.830525 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:09:59.831221 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.831236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.832894 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.832903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.835304 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:09:59.835320 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:09:59.837234 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:09:59.837240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:09:59.837444 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:09:59.837516 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 11:09:59.837661 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 11:09:59.837894 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11987,"mutex_wait_us":45,"thread_start_us":11383,"threads_started":1}
I20260925 11:09:59.843266 23073 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (DEFAULT_TABLE table=test-table [id=38ab1cb047474066914d27ff94e9eb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:59.843465 23207 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (DEFAULT_TABLE table=test-table [id=38ab1cb047474066914d27ff94e9eb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:59.843465 22939 tablet_service.cc:1467] Processing CreateTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (DEFAULT_TABLE table=test-table [id=38ab1cb047474066914d27ff94e9eb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:09:59.844635 23073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:59.844866 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:59.844868 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:09:59.856966 23291 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Bootstrap starting.
I20260925 11:09:59.859843 23292 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Bootstrap starting.
I20260925 11:09:59.861011 23291 tablet_bootstrap.cc:654] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:59.862429 23291 log.cc:824] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:59.863720 23292 tablet_bootstrap.cc:654] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:59.865100 23292 log.cc:824] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:59.866272 23291 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: No bootstrap required, opened a new log
I20260925 11:09:59.866621 23291 ts_tablet_manager.cc:1397] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260925 11:09:59.868364 23292 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: No bootstrap required, opened a new log
I20260925 11:09:59.868655 23292 ts_tablet_manager.cc:1397] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260925 11:09:59.864104 23293 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Bootstrap starting.
I20260925 11:09:59.876332 23292 raft_consensus.cc:348] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.876752 23292 raft_consensus.cc:374] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:59.876873 23292 raft_consensus.cc:729] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95f155cbfca486b92797d11bf397ca5, State: Initialized, Role: FOLLOWER
I20260925 11:09:59.876919 23291 raft_consensus.cc:348] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.877332 23291 raft_consensus.cc:374] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:59.877434 23291 raft_consensus.cc:729] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06d1b0593db44e690ccc8b6d567b3f7, State: Initialized, Role: FOLLOWER
I20260925 11:09:59.877571 23292 consensus_queue.cc:260] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.878125 23293 tablet_bootstrap.cc:654] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Neither blocks nor log segments found. Creating new log.
I20260925 11:09:59.877990 23291 consensus_queue.cc:260] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.879556 23293 log.cc:824] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:09:59.879827 23292 ts_tablet_manager.cc:1428] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260925 11:09:59.886695 23138 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:09:59.887415 23291 ts_tablet_manager.cc:1428] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
I20260925 11:09:59.888706 23293 tablet_bootstrap.cc:492] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: No bootstrap required, opened a new log
I20260925 11:09:59.889039 23293 ts_tablet_manager.cc:1397] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.006s
W20260925 11:09:59.890225 23005 tablet.cc:2369] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:09:59.897686 23293 raft_consensus.cc:348] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.898211 23293 raft_consensus.cc:374] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:09:59.898344 23293 raft_consensus.cc:729] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3d18c81bca4b888b9fc10345a7c29f, State: Initialized, Role: FOLLOWER
I20260925 11:09:59.899062 23293 consensus_queue.cc:260] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:09:59.900146 23272 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:09:59.900656 23293 ts_tablet_manager.cc:1428] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260925 11:09:59.939234 23139 tablet.cc:2369] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:09:59.987972 23273 tablet.cc:2369] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:10:00.200585 23296 raft_consensus.cc:482] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:10:00.201089 23296 raft_consensus.cc:504] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:00.202991 23296 leader_election.cc:288] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243), c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
I20260925 11:10:00.210599 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.203440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:49901 (local address 127.8.181.3:45243)
0925 11:10:00.203681 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.203690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.204515 (+ 825us) server_negotiation.cc:407] Connection header received
0925 11:10:00.204695 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.204699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.204760 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.204831 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:00.205610 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.206401 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.208629 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.209477 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.209577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.209667 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.210132 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.210265 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:10:00.210781 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.203351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:34383 (local address 127.8.181.1:49611)
0925 11:10:00.204313 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:00.204330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:00.204523 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:00.205213 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:00.205218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:00.205241 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:00.205659 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:00.205673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.206885 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.206889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:00.208518 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:00.208563 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.209862 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.209866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:00.209988 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.210010 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 11:10:00.210123 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 11:10:00.210265 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":40,"thread_start_us":606,"threads_started":1}
I20260925 11:10:00.210822 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.203360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45243 (local address 127.8.181.1:49901)
0925 11:10:00.204422 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:00.204437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:00.204549 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:00.204915 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:00.204918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:00.204938 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:00.205435 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:00.205447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.206563 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.206570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:00.208444 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:00.208459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.209708 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.209717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:00.209935 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.209985 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 11:10:00.210067 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 11:10:00.210255 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":64,"thread_start_us":529,"threads_started":1}
I20260925 11:10:00.210618 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.203481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:49611 (local address 127.8.181.2:34383)
0925 11:10:00.203745 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.203753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.204442 (+ 689us) server_negotiation.cc:407] Connection header received
0925 11:10:00.204985 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.204989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.205041 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.205155 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:00.205848 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.206744 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.208718 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.209634 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.209787 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.209900 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.210177 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.210307 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:10:00.211930 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2c9ee082d24f3c8c379f60fdf1edf8" candidate_uuid: "a95f155cbfca486b92797d11bf397ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" is_pre_election: true
I20260925 11:10:00.212139 23093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2c9ee082d24f3c8c379f60fdf1edf8" candidate_uuid: "a95f155cbfca486b92797d11bf397ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" is_pre_election: true
I20260925 11:10:00.212502 23227 raft_consensus.cc:2454] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95f155cbfca486b92797d11bf397ca5 in term 0.
I20260925 11:10:00.212666 23093 raft_consensus.cc:2454] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a95f155cbfca486b92797d11bf397ca5 in term 0.
I20260925 11:10:00.213264 22893 leader_election.cc:302] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3d18c81bca4b888b9fc10345a7c29f, a95f155cbfca486b92797d11bf397ca5; no voters:
I20260925 11:10:00.213707 23296 raft_consensus.cc:2781] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:10:00.213908 23296 raft_consensus.cc:482] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:10:00.213994 23296 raft_consensus.cc:3037] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:00.216216 23296 raft_consensus.cc:504] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:00.216830 23296 leader_election.cc:288] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243), c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
I20260925 11:10:00.217437 23227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2c9ee082d24f3c8c379f60fdf1edf8" candidate_uuid: "a95f155cbfca486b92797d11bf397ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
I20260925 11:10:00.217475 23093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b2c9ee082d24f3c8c379f60fdf1edf8" candidate_uuid: "a95f155cbfca486b92797d11bf397ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
I20260925 11:10:00.217664 23227 raft_consensus.cc:3037] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:00.217694 23093 raft_consensus.cc:3037] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:00.220008 23093 raft_consensus.cc:2454] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95f155cbfca486b92797d11bf397ca5 in term 1.
I20260925 11:10:00.220021 23227 raft_consensus.cc:2454] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a95f155cbfca486b92797d11bf397ca5 in term 1.
I20260925 11:10:00.220556 22892 leader_election.cc:302] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a95f155cbfca486b92797d11bf397ca5, c06d1b0593db44e690ccc8b6d567b3f7; no voters:
I20260925 11:10:00.220983 23296 raft_consensus.cc:2781] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:10:00.221514 23296 raft_consensus.cc:686] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 1 LEADER]: Becoming Leader. State: Replica: a95f155cbfca486b92797d11bf397ca5, State: Running, Role: LEADER
I20260925 11:10:00.222066 23296 consensus_queue.cc:237] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:00.228073 22814 catalog_manager.cc:5495] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a95f155cbfca486b92797d11bf397ca5 (127.8.181.1). New cstate: current_term: 1 leader_uuid: "a95f155cbfca486b92797d11bf397ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } health_report { overall_health: UNKNOWN } } }
I20260925 11:10:00.289628 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.283606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.8.181.1:34751)
0925 11:10:00.283851 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.283855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.283876 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:10:00.283972 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.283975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.284055 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.284220 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:00.284617 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.285431 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.287104 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.287549 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.287659 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.288214 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:00.289003 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:10:00.289131 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.289322 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.289403 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:10:00.298444 23093 raft_consensus.cc:1261] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Refusing update from remote peer a95f155cbfca486b92797d11bf397ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:00.298558 23227 raft_consensus.cc:1261] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Refusing update from remote peer a95f155cbfca486b92797d11bf397ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:00.299726 23302 consensus_queue.cc:1035] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:00.300139 23296 consensus_queue.cc:1035] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:00.313387 23305 mvcc.cc:205] Tried to move back new op lower bound from 7333210522811510784 to 7333210522514571264. Current Snapshot: MvccSnapshot[applied={T|T < 7333210522811510784}]
I20260925 11:10:00.328569 22815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:00.328922 22815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:00.332636 22815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Sending DeleteTablet for 3 replicas of tablet 5b2c9ee082d24f3c8c379f60fdf1edf8
I20260925 11:10:00.333930 23207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:00 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53472
I20260925 11:10:00.333933 22939 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:00 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:50640
I20260925 11:10:00.334209 23073 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:00 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:33290
I20260925 11:10:00.335379 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22876
I20260925 11:10:00.345432 23315 tablet_replica.cc:330] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: stopping tablet replica
I20260925 11:10:00.346105 23315 raft_consensus.cc:2229] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:00.348620 23315 raft_consensus.cc:2258] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:00.349223 23316 tablet_replica.cc:330] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: stopping tablet replica
I20260925 11:10:00.349812 23316 raft_consensus.cc:2229] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:10:00.350575 23316 raft_consensus.cc:2258] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:00.351780 23315 ts_tablet_manager.cc:1905] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:00.352891 23316 ts_tablet_manager.cc:1905] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:00.361040 23317 tablet_replica.cc:330] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: stopping tablet replica
I20260925 11:10:00.361202 23316 ts_tablet_manager.cc:1918] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:00.361439 23316 log.cc:1196] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/5b2c9ee082d24f3c8c379f60fdf1edf8
I20260925 11:10:00.361685 23315 ts_tablet_manager.cc:1918] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:00.361869 23315 log.cc:1196] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/5b2c9ee082d24f3c8c379f60fdf1edf8
I20260925 11:10:00.362054 23316 ts_tablet_manager.cc:1939] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P a95f155cbfca486b92797d11bf397ca5: Deleting consensus metadata
I20260925 11:10:00.362443 23315 ts_tablet_manager.cc:1939] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting consensus metadata
I20260925 11:10:00.362046 23317 raft_consensus.cc:2229] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:00.363262 23317 raft_consensus.cc:2258] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:00.364131 22799 catalog_manager.cc:4841] TS a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751): tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (table test-table [id=38ab1cb047474066914d27ff94e9eb28]) successfully deleted
I20260925 11:10:00.364526 22800 catalog_manager.cc:4841] TS 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243): tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (table test-table [id=38ab1cb047474066914d27ff94e9eb28]) successfully deleted
I20260925 11:10:00.367879 23317 ts_tablet_manager.cc:1905] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 11:10:00 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:10:00.375471 23317 ts_tablet_manager.cc:1918] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:00.375663 23317 log.cc:1196] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/5b2c9ee082d24f3c8c379f60fdf1edf8
I20260925 11:10:00.376215 23317 ts_tablet_manager.cc:1939] T 5b2c9ee082d24f3c8c379f60fdf1edf8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting consensus metadata
I20260925 11:10:00.377137 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.337186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.8.181.1:34751)
0925 11:10:00.337402 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.337408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.350814 (+ 13406us) server_negotiation.cc:407] Connection header received
0925 11:10:00.351375 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.351381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.351473 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.351565 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:00.353954 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.355306 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.363864 (+ 8558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.364606 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.364719 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.369872 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:00.369887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:00.369901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:00.369946 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:00.373496 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.374541 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.374556 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.374559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.374711 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.375192 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.375196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.375201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.375641 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:00.375900 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.376479 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.376737 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:10:00.377885 22999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:00.378235 22799 catalog_manager.cc:4841] TS c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383): tablet 5b2c9ee082d24f3c8c379f60fdf1edf8 (table test-table [id=38ab1cb047474066914d27ff94e9eb28]) successfully deleted
I20260925 11:10:00.481098 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22876
I20260925 11:10:00.502672 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23010
Sep 25 11:10:00 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:10:00.519312 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.503638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.8.181.2:34383)
0925 11:10:00.503820 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.503825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.503844 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:10:00.503999 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.504003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.504073 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.504164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:00.505464 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.506300 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.507618 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.508032 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.508147 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.512899 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:00.512916 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:00.512936 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:00.512984 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:00.516238 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.517091 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.517100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.517103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.517246 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.517621 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.517624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.517626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.518062 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:00.518329 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.518651 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.518879 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 11:10:00.520098 23133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:00.625447 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23010
I20260925 11:10:00.643118 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23144
Sep 25 11:10:00 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:10:00.660424 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:00.644041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.8.181.3:45243)
0925 11:10:00.644253 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:00.644258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:00.644481 (+ 223us) server_negotiation.cc:407] Connection header received
0925 11:10:00.644621 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:00.644625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:00.644698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:00.644801 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:00.646314 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.647202 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.648718 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:00.649162 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:00.649283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:00.654262 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:00.654279 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:00.654296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:00.654339 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:00.657332 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.658217 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.658225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.658231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.658394 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:00.658985 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:00.658989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:00.658991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:00.659380 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:00.659621 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:00.659970 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 11:10:00.660151 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 11:10:00.661285 23267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:00.752537 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23144
I20260925 11:10:00.769614 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 22783
I20260925 11:10:00.770773 22845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:00.884122 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 22783
I20260925 11:10:00.908700 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--webserver_interface=127.8.181.62
--webserver_port=44841
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:01.257596 23323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:01.257989 23323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:01.258160 23323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:01.265686 23323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:10:01.265872 23323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:01.265988 23323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:01.266095 23323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:10:01.266185 23323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:10:01.276564 23323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=44841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:01.277990 23323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:01.279299 23323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:01.302440 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:01.302561 23329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:01.302546 23332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:01.303426 23323 server_base.cc:1034] running on GCE node
I20260925 11:10:01.304373 23323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:01.306155 23323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:01.307417 23323 hybrid_clock.cc:648] HybridClock initialized: now 1790334601307327 us; error 64 us; skew 500 ppm
I20260925 11:10:01.307906 23323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:01 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334601, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:01.317651 23323 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:10:01.319674 23323 webserver.cc:460] Webserver started at http://127.8.181.62:44841/ using document root <none> and password file <none>
I20260925 11:10:01.320549 23323 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:01.320736 23323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:01.327343 23323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:10:01.330833 23339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:01.332504 23323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260925 11:10:01.332739 23323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a4de74eae4a8417cbfed00bc2e21ba61"
format_stamp: "Formatted at 2026-09-25 11:09:57 on dist-test-slave-6042"
server_key: "18097937c1c3b85937d021604719dffd"
server_key_iv: "897696643e0821a9e15840b9cffbb5b9"
server_key_version: "encryptionkey@0"
I20260925 11:10:01.333931 23323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:01.348608 23323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:01.353466 23323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:01.353773 23323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:01.376546 23323 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:44335
I20260925 11:10:01.376588 23390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:44335 every 8 connection(s)
I20260925 11:10:01.378320 23323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:10:01.384595 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23323
I20260925 11:10:01.385433 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:34751
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=36299
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:10:01.386875 23391 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:10:01.390637 23391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap starting.
I20260925 11:10:01.411520 23391 log.cc:824] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:01.424510 23391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:10:01.425130 23391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap complete.
I20260925 11:10:01.434719 23391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:01.435654 23391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Initialized, Role: FOLLOWER
I20260925 11:10:01.436333 23391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:01.436602 23391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:10:01.436746 23391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:10:01.436939 23391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:10:01.441553 23391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:01.442260 23391 leader_election.cc:302] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4de74eae4a8417cbfed00bc2e21ba61; no voters:
I20260925 11:10:01.443011 23391 leader_election.cc:288] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 11:10:01.443166 23395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 11:10:01.444005 23395 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 LEADER]: Becoming Leader. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Running, Role: LEADER
I20260925 11:10:01.444785 23395 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:01.446187 23391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:10:01.447697 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4de74eae4a8417cbfed00bc2e21ba61. Latest consensus state: current_term: 2 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:10:01.447662 23397 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:10:01.448307 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:01.448315 23397 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:01.453382 23400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:10:01.464023 23400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:10:01.466158 23400 catalog_manager.cc:1259] Loaded cluster ID: 40156d3ba3a54071af075d6462fa4341
I20260925 11:10:01.466378 23400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:10:01.470892 23400 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 11:10:01.489599 23400 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 11:10:01.495394 23412 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 11:10:01.500411 23400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:10:01.502266 23400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Loaded TSK: 0
I20260925 11:10:01.504009 23400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 11:10:01.754864 23393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:01.755162 23393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:01.755314 23393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:01.762907 23393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:01.763031 23393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:01.763211 23393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:10:01.773502 23393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:34751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=36299
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:01.774941 23393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:01.776377 23393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:01.791473 23419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:01.791491 23418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:01.791733 23421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:01.793411 23393 server_base.cc:1034] running on GCE node
I20260925 11:10:01.794088 23393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:01.795220 23393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:01.796482 23393 hybrid_clock.cc:648] HybridClock initialized: now 1790334601796421 us; error 54 us; skew 500 ppm
I20260925 11:10:01.796842 23393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:01 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334601, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:01.800242 23393 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:10:01.801868 23393 webserver.cc:460] Webserver started at http://127.8.181.1:36299/ using document root <none> and password file <none>
I20260925 11:10:01.802724 23393 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:01.802888 23393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:01.809160 23393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 11:10:01.812311 23428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:01.813934 23393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:10:01.814149 23393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "a95f155cbfca486b92797d11bf397ca5"
format_stamp: "Formatted at 2026-09-25 11:09:58 on dist-test-slave-6042"
server_key: "13a001651154365df0658df18cf559e5"
server_key_iv: "a5e3d8d5b51179851323b5a93be4bfe0"
server_key_version: "encryptionkey@0"
I20260925 11:10:01.815297 23393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:01.848610 23393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:01.853569 23393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:01.853893 23393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:01.855043 23393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:01.856498 23393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:01.856632 23393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:01.856765 23393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:01.856855 23393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:01.891750 23393 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34751
I20260925 11:10:01.891849 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34751 every 8 connection(s)
I20260925 11:10:01.893222 23393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:10:01.901072 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23393
I20260925 11:10:01.901695 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:34383
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=38023
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:10:01 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334601, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:01.919639 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:01.896201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:39709 (local address 127.8.181.62:44335)
0925 11:10:01.896830 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:01.896841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:01.897252 (+ 411us) server_negotiation.cc:407] Connection header received
0925 11:10:01.899559 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:01.899572 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:01.899911 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:01.900192 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:01.901874 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:01.903985 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:01.905686 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:01.906911 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:01.907102 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:01.911142 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:01.911151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:01.911168 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:01.911214 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:01.914452 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:01.915434 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:01.915441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:01.915446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:01.915665 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:01.916173 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:01.916179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:01.916182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:01.916717 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:01.916935 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:01.917744 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 11:10:01.918165 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260925 11:10:01.919628 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:01.896274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:39709)
0925 11:10:01.897071 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:01.897121 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:01.899028 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:01.900366 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:01.900373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:01.900934 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:01.901604 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:01.901623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:01.904189 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:01.904193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:01.905520 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:01.905532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:01.907194 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:01.907199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:01.907390 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:01.908152 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:01.908181 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:01.910792 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:01.914690 (+ 3898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:01.914700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:01.914716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:01.915214 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:01.915820 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:01.915825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:01.915828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:01.916034 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:01.916889 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:01.916897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:01.917246 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 11:10:01.917537 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 11:10:01.918164 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":569,"threads_started":1}
I20260925 11:10:01.923097 23541 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
I20260925 11:10:01.923460 23541 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:01.924222 23541 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
I20260925 11:10:01.926872 23356 ts_manager.cc:194] Registered new tserver with Master: a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751)
I20260925 11:10:01.929018 23356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:39709
W20260925 11:10:02.243376 23545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:02.243687 23545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:02.243784 23545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:02.251420 23545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:02.251549 23545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:02.251663 23545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:10:02.261719 23545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:34383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=38023
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:02.263109 23545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:02.264420 23545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:02.278369 23551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:02.278456 23550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:02.278518 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:02.279299 23545 server_base.cc:1034] running on GCE node
I20260925 11:10:02.279944 23545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:02.280887 23545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:02.282117 23545 hybrid_clock.cc:648] HybridClock initialized: now 1790334602282076 us; error 34 us; skew 500 ppm
I20260925 11:10:02.282577 23545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:02 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:02.286197 23545 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:10:02.287925 23545 webserver.cc:460] Webserver started at http://127.8.181.2:38023/ using document root <none> and password file <none>
I20260925 11:10:02.288604 23545 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:02.288718 23545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:02.294620 23545 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 11:10:02.297540 23560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.299243 23545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:10:02.299396 23545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "c7a6ddfe52111dd57cf63e94e496346e"
server_key_iv: "11567058c1d0e7b55e2bb049089d975a"
server_key_version: "encryptionkey@0"
I20260925 11:10:02.300367 23545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:02.329768 23545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:02.334285 23545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:02.334604 23545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:02.335568 23545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:02.336951 23545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:02.337036 23545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.337109 23545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:02.337198 23545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.373508 23545 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:34383
I20260925 11:10:02.373600 23672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:34383 every 8 connection(s)
I20260925 11:10:02.375216 23545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:10:02.378508 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23545
I20260925 11:10:02.379437 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:45243
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=34071
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:10:02 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:02.409096 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.379731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:50487 (local address 127.8.181.62:44335)
0925 11:10:02.379963 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.379970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.391441 (+ 11471us) server_negotiation.cc:407] Connection header received
0925 11:10:02.393545 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.393551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.393674 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.393793 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:02.395613 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.396330 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.397945 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.398449 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.398646 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.402371 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:02.402376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:02.402379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:02.402416 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:02.405630 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.406435 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.406441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.406445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.406648 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.407106 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.407111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.407115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.407416 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:02.407573 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.408454 (+ 881us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.408723 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 11:10:02.410332 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.379069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:50487)
0925 11:10:02.391225 (+ 12156us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.391288 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.393232 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.394102 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.394111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.394748 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:02.395427 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.395445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.396519 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.396523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.397776 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.397788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.398742 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.398750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.398999 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.399741 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:02.399766 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:02.402119 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:02.405803 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:02.405813 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:02.405831 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:02.406291 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:02.406763 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:02.406768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:02.406772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:02.406979 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:02.407556 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:02.407564 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:02.407934 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 11:10:02.408343 (+ 409us) client_negotiation.cc:240] Negotiation successful
0925 11:10:02.408924 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11823,"thread_start_us":11578,"threads_started":1}
I20260925 11:10:02.412005 23673 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
I20260925 11:10:02.412341 23673 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:02.413091 23673 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
I20260925 11:10:02.414638 23356 ts_manager.cc:194] Registered new tserver with Master: c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
I20260925 11:10:02.415457 23356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:50487
W20260925 11:10:02.731854 23676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:02.732214 23676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:02.732395 23676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:02.740180 23676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:02.740357 23676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:02.740542 23676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:10:02.750839 23676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:45243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=34071
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:02.752195 23676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:02.753559 23676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:02.768067 23684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:02.768142 23682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:02.768354 23681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:02.774044 23676 server_base.cc:1034] running on GCE node
I20260925 11:10:02.774869 23676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:02.776003 23676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:02.777295 23676 hybrid_clock.cc:648] HybridClock initialized: now 1790334602777220 us; error 63 us; skew 500 ppm
I20260925 11:10:02.777683 23676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:02 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:02.781378 23676 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:10:02.783516 23676 webserver.cc:460] Webserver started at http://127.8.181.3:34071/ using document root <none> and password file <none>
I20260925 11:10:02.784353 23676 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:02.784528 23676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:02.790449 23676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:10:02.793577 23691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.795307 23676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 11:10:02.795508 23676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "8d1aeafaf9e812a63e1234c1409379fd"
server_key_iv: "f2802d9e928450e200b7363d2ea9ad26"
server_key_version: "encryptionkey@0"
I20260925 11:10:02.796509 23676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:02.809113 23676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:02.813591 23676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:02.813833 23676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:02.814900 23676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:02.816284 23676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:02.816372 23676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.816445 23676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:02.816499 23676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:02.850647 23676 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45243
I20260925 11:10:02.850698 23803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45243 every 8 connection(s)
I20260925 11:10:02.852155 23676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:10:02.860409 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23676
Sep 25 11:10:02 dist-test-slave-6042 krb5kdc[22770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:02.878129 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.861533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.8.181.62:44335)
0925 11:10:02.862269 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.862275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.862293 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:10:02.862346 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.862349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.862403 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.862674 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:02.864334 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.865253 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.866943 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.867411 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.867550 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.870729 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:02.870740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:02.870744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:02.870804 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:02.875206 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.876041 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.876049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.876052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.876177 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.876589 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.876596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.876599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.876869 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:02.877138 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.877455 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.877737 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":597,"thread_start_us":444,"threads_started":1}
I20260925 11:10:02.878162 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.855432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:48821 (local address 127.8.181.62:44335)
0925 11:10:02.855681 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.855688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.857141 (+ 1453us) server_negotiation.cc:407] Connection header received
0925 11:10:02.859207 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.859212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.859295 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.859385 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:02.861170 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.862564 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.864367 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.864937 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.865162 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.869591 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:02.869597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:02.869601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:02.869639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:02.874404 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.875444 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.875449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.875453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.875590 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:02.876062 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:02.876067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:02.876071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:02.876393 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:02.876554 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.877325 (+ 771us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.877569 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:10:02.879272 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.856099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:48821)
0925 11:10:02.856963 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.857012 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.858841 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.859615 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.859624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.860257 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:02.860919 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.860938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.862751 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.862758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.864195 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.864208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.865271 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.865283 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.865563 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.866373 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:02.866418 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:02.869213 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:02.874677 (+ 5464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:02.874688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:02.874718 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:02.875236 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:02.875709 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:02.875714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:02.875717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:02.875939 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:02.876536 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:02.876545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:02.876923 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 11:10:02.877228 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 11:10:02.877818 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":344,"threads_started":1}
I20260925 11:10:02.880931 23804 heartbeater.cc:347] Connected to a master server at 127.8.181.62:44335
I20260925 11:10:02.881258 23804 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:02.881949 23804 heartbeater.cc:511] Master 127.8.181.62:44335 requested a full tablet report, sending...
I20260925 11:10:02.883381 23356 ts_manager.cc:194] Registered new tserver with Master: 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243)
I20260925 11:10:02.884202 23356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:48821
I20260925 11:10:02.893690 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:10:02.901201 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.894737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.8.181.62:44335)
0925 11:10:02.894997 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.895007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.895442 (+ 435us) server_negotiation.cc:407] Connection header received
0925 11:10:02.895647 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.895651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.895801 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.895939 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:02.896442 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.897169 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.898823 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.899348 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.899450 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.900008 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:02.900574 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:10:02.900682 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.900814 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.900887 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 11:10:02.902287 23356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:10:02.910082 23356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:10:02.932493 23541 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:10:02.935168 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.922717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45243 (local address 127.0.0.1:42058)
0925 11:10:02.923698 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.923748 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.923988 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.927281 (+ 3293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.927289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.927335 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:02.927988 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.928002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.930252 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.930258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.932689 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.932708 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.934348 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.934355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.934593 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.934637 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 11:10:02.934725 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 11:10:02.934865 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":57,"thread_start_us":947,"threads_started":1}
I20260925 11:10:02.936314 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.923085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.8.181.3:45243)
0925 11:10:02.926565 (+ 3480us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.926578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.926616 (+ 38us) server_negotiation.cc:407] Connection header received
0925 11:10:02.926724 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.926732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.927009 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.928136 (+ 1127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:02.928469 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.930090 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.932900 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.934111 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.935450 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.935696 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.935788 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.935970 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3273,"thread_start_us":3128,"threads_started":1}
I20260925 11:10:02.936900 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.925517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:34383 (local address 127.0.0.1:53600)
0925 11:10:02.927786 (+ 2269us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.927805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.927955 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.928783 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.928787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.928814 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:02.929231 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.929247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.931264 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.931274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.934028 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.934040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.935918 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.935924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.936093 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.936130 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 11:10:02.936213 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 11:10:02.936362 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"thread_start_us":1961,"threads_started":1}
I20260925 11:10:02.937230 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.926166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.8.181.2:34383)
0925 11:10:02.927250 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.927262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.928209 (+ 947us) server_negotiation.cc:407] Connection header received
0925 11:10:02.928313 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.928319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.928541 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.928760 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:02.929464 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.931097 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.934209 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.935701 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.935845 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.936414 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.936482 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.936641 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":795,"threads_started":1}
I20260925 11:10:02.937925 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.922924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34751 (local address 127.0.0.1:52680)
0925 11:10:02.925742 (+ 2818us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.925766 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.925920 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.927487 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.927493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.927518 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:02.927998 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.928008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.930100 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.930106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.933118 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.933133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.937020 (+ 3887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.937027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.937198 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.937240 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 11:10:02.937349 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 11:10:02.937519 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2661,"mutex_wait_us":76,"thread_start_us":2192,"threads_started":1}
I20260925 11:10:02.938853 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.923455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.8.181.1:34751)
0925 11:10:02.926932 (+ 3477us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.926939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.926965 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:10:02.927058 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.927064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.927275 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.928220 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:02.928474 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.929928 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.934038 (+ 4110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.935916 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.936034 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:02.936231 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:02.938330 (+ 2099us) server_negotiation.cc:299] Negotiation successful
0925 11:10:02.938538 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3313,"thread_start_us":3168,"threads_started":1}
I20260925 11:10:02.940245 23739 tablet_service.cc:1467] Processing CreateTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 (DEFAULT_TABLE table=test-table [id=13d59b6327954c1b8bcc557983805052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:02.941972 23739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fc488b3f3e46c790a6e8d55b0364e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:02.942147 23476 tablet_service.cc:1467] Processing CreateTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 (DEFAULT_TABLE table=test-table [id=13d59b6327954c1b8bcc557983805052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:02.943481 23476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fc488b3f3e46c790a6e8d55b0364e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:02.944363 23608 tablet_service.cc:1467] Processing CreateTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 (DEFAULT_TABLE table=test-table [id=13d59b6327954c1b8bcc557983805052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:02.945668 23608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87fc488b3f3e46c790a6e8d55b0364e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:02.956743 23820 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Bootstrap starting.
I20260925 11:10:02.959470 23821 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Bootstrap starting.
I20260925 11:10:02.960990 23822 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Bootstrap starting.
I20260925 11:10:02.960727 23820 tablet_bootstrap.cc:654] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:02.963073 23820 log.cc:824] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:02.963564 23821 tablet_bootstrap.cc:654] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:02.964602 23822 tablet_bootstrap.cc:654] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:02.964871 23821 log.cc:824] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:02.966022 23822 log.cc:824] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:02.966935 23820 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: No bootstrap required, opened a new log
I20260925 11:10:02.967237 23820 ts_tablet_manager.cc:1397] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 11:10:02.967952 23821 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: No bootstrap required, opened a new log
I20260925 11:10:02.968226 23821 ts_tablet_manager.cc:1397] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 11:10:02.969048 23822 tablet_bootstrap.cc:492] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: No bootstrap required, opened a new log
I20260925 11:10:02.969302 23822 ts_tablet_manager.cc:1397] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260925 11:10:02.976012 23821 raft_consensus.cc:348] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.976600 23821 raft_consensus.cc:374] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:02.976704 23821 raft_consensus.cc:729] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3d18c81bca4b888b9fc10345a7c29f, State: Initialized, Role: FOLLOWER
I20260925 11:10:02.977396 23821 consensus_queue.cc:260] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.978648 23804 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:10:02.978407 23820 raft_consensus.cc:348] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.978888 23820 raft_consensus.cc:374] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:02.978979 23820 raft_consensus.cc:729] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95f155cbfca486b92797d11bf397ca5, State: Initialized, Role: FOLLOWER
I20260925 11:10:02.979529 23821 ts_tablet_manager.cc:1428] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260925 11:10:02.979588 23820 consensus_queue.cc:260] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.980770 23822 raft_consensus.cc:348] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.981364 23822 raft_consensus.cc:374] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:02.981483 23822 raft_consensus.cc:729] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06d1b0593db44e690ccc8b6d567b3f7, State: Initialized, Role: FOLLOWER
I20260925 11:10:02.982560 23822 consensus_queue.cc:260] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.986516 23826 raft_consensus.cc:482] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:10:02.987207 23826 raft_consensus.cc:504] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:02.989241 23820 ts_tablet_manager.cc:1428] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Time spent starting tablet: real 0.022s user 0.014s sys 0.007s
I20260925 11:10:02.990646 23826 leader_election.cc:288] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751), c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
I20260925 11:10:02.991086 23673 heartbeater.cc:503] Master 127.8.181.62:44335 was elected leader, sending a full tablet report...
I20260925 11:10:02.996042 23822 ts_tablet_manager.cc:1428] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Time spent starting tablet: real 0.027s user 0.020s sys 0.002s
I20260925 11:10:03.000921 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.992550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52509 (local address 127.8.181.1:34751)
0925 11:10:02.993090 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.993099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.993125 (+ 26us) server_negotiation.cc:407] Connection header received
0925 11:10:02.993210 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.993218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.993310 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.993426 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:02.994453 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.995859 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.998657 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.999954 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.000101 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.000220 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.000528 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.000663 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260925 11:10:03.000921 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.991291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34751 (local address 127.8.181.3:52509)
0925 11:10:02.992017 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.992038 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.992148 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.993921 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.993929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.993970 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:02.994245 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.994259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.996180 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.996185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:02.998475 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.998509 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.000119 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.000123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:03.000370 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.000396 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 11:10:03.000473 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 11:10:03.000572 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":1}
I20260925 11:10:03.001916 23496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fc488b3f3e46c790a6e8d55b0364e8" candidate_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95f155cbfca486b92797d11bf397ca5" is_pre_election: true
I20260925 11:10:03.002518 23496 raft_consensus.cc:2454] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3d18c81bca4b888b9fc10345a7c29f in term 0.
I20260925 11:10:03.003437 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.992856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:39641 (local address 127.8.181.2:34383)
0925 11:10:02.993171 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:02.993180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:02.996080 (+ 2900us) server_negotiation.cc:407] Connection header received
0925 11:10:02.997139 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:02.997144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:02.997226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:02.997330 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:02.998102 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.999063 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.001463 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.002340 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.002448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.002592 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.003010 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.003159 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 11:10:03.003446 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:02.991610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:34383 (local address 127.8.181.3:39641)
0925 11:10:02.995955 (+ 4345us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:02.995982 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:02.996977 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:02.997370 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:02.997375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:02.997401 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:02.997876 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:02.997893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:02.999233 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:02.999239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:03.001265 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:03.001279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.002570 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.002576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:03.002770 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.002809 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 11:10:03.002914 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 11:10:03.003074 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4088,"thread_start_us":3906,"threads_started":1}
I20260925 11:10:03.003327 23692 leader_election.cc:302] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3d18c81bca4b888b9fc10345a7c29f, a95f155cbfca486b92797d11bf397ca5; no voters:
I20260925 11:10:03.004001 23828 raft_consensus.cc:2781] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:10:03.004160 23828 raft_consensus.cc:482] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:10:03.004330 23828 raft_consensus.cc:3037] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:03.004938 23628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fc488b3f3e46c790a6e8d55b0364e8" candidate_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" is_pre_election: true
I20260925 11:10:03.005401 23628 raft_consensus.cc:2454] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d3d18c81bca4b888b9fc10345a7c29f in term 0.
I20260925 11:10:03.007084 23828 raft_consensus.cc:504] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:03.007781 23828 leader_election.cc:288] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [CANDIDATE]: Term 1 election: Requested vote from peers a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751), c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383)
I20260925 11:10:03.008340 23628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fc488b3f3e46c790a6e8d55b0364e8" candidate_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
I20260925 11:10:03.008348 23496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87fc488b3f3e46c790a6e8d55b0364e8" candidate_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95f155cbfca486b92797d11bf397ca5"
I20260925 11:10:03.008577 23496 raft_consensus.cc:3037] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:03.008584 23628 raft_consensus.cc:3037] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:03.011317 23496 raft_consensus.cc:2454] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3d18c81bca4b888b9fc10345a7c29f in term 1.
I20260925 11:10:03.011309 23628 raft_consensus.cc:2454] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d3d18c81bca4b888b9fc10345a7c29f in term 1.
I20260925 11:10:03.011780 23692 leader_election.cc:302] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3d18c81bca4b888b9fc10345a7c29f, a95f155cbfca486b92797d11bf397ca5; no voters:
I20260925 11:10:03.012122 23828 raft_consensus.cc:2781] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:10:03.012346 23828 raft_consensus.cc:686] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 LEADER]: Becoming Leader. State: Replica: 3d3d18c81bca4b888b9fc10345a7c29f, State: Running, Role: LEADER
I20260925 11:10:03.012856 23828 consensus_queue.cc:237] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } }
I20260925 11:10:03.018105 23355 catalog_manager.cc:5495] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3). New cstate: current_term: 1 leader_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3d18c81bca4b888b9fc10345a7c29f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 } health_report { overall_health: UNKNOWN } } }
I20260925 11:10:03.043421 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:03.037828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.8.181.3:45243)
0925 11:10:03.038045 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:03.038051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:03.038070 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:10:03.038317 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:03.038321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:03.038401 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:03.038616 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:03.039025 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.039743 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.041173 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.041579 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.041716 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.042227 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:03.042831 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:10:03.042990 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.043070 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.043152 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:10:03.052598 23628 raft_consensus.cc:1261] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Refusing update from remote peer 3d3d18c81bca4b888b9fc10345a7c29f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:03.052596 23496 raft_consensus.cc:1261] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Refusing update from remote peer 3d3d18c81bca4b888b9fc10345a7c29f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:03.053769 23828 consensus_queue.cc:1035] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06d1b0593db44e690ccc8b6d567b3f7" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 34383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:03.054121 23826 consensus_queue.cc:1035] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95f155cbfca486b92797d11bf397ca5" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:03.067883 23834 mvcc.cc:205] Tried to move back new op lower bound from 7333210534091624448 to 7333210533944856576. Current Snapshot: MvccSnapshot[applied={T|T < 7333210534091624448}]
I20260925 11:10:03.070551 23835 mvcc.cc:205] Tried to move back new op lower bound from 7333210534091624448 to 7333210533944856576. Current Snapshot: MvccSnapshot[applied={T|T < 7333210534091624448}]
I20260925 11:10:03.083602 23355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:03.083915 23355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:03.087535 23355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Sending DeleteTablet for 3 replicas of tablet 87fc488b3f3e46c790a6e8d55b0364e8
I20260925 11:10:03.088795 23739 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:03 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:42058
I20260925 11:10:03.088795 23476 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:03 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:52680
I20260925 11:10:03.089725 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23393
I20260925 11:10:03.090070 23608 tablet_service.cc:1514] Processing DeleteTablet for tablet 87fc488b3f3e46c790a6e8d55b0364e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:03 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:53600
I20260925 11:10:03.090242 23844 tablet_replica.cc:330] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: stopping tablet replica
I20260925 11:10:03.090883 23844 raft_consensus.cc:2229] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:03.089767 23843 tablet_replica.cc:330] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: stopping tablet replica
I20260925 11:10:03.091460 23843 raft_consensus.cc:2229] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:10:03.092249 23843 raft_consensus.cc:2258] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:03.092803 23844 raft_consensus.cc:2258] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:03.095453 23843 ts_tablet_manager.cc:1905] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:03.101449 23845 tablet_replica.cc:330] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: stopping tablet replica
I20260925 11:10:03.101966 23845 raft_consensus.cc:2229] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:03.102563 23845 raft_consensus.cc:2258] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:03.104287 23843 ts_tablet_manager.cc:1918] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:03.104542 23843 log.cc:1196] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/87fc488b3f3e46c790a6e8d55b0364e8
I20260925 11:10:03.105224 23843 ts_tablet_manager.cc:1939] T 87fc488b3f3e46c790a6e8d55b0364e8 P 3d3d18c81bca4b888b9fc10345a7c29f: Deleting consensus metadata
I20260925 11:10:03.105762 23844 ts_tablet_manager.cc:1905] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:03.107281 23341 catalog_manager.cc:4841] TS 3d3d18c81bca4b888b9fc10345a7c29f (127.8.181.3:45243): tablet 87fc488b3f3e46c790a6e8d55b0364e8 (table test-table [id=13d59b6327954c1b8bcc557983805052]) successfully deleted
I20260925 11:10:03.110994 23845 ts_tablet_manager.cc:1905] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:03.113466 23844 ts_tablet_manager.cc:1918] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:03.113660 23844 log.cc:1196] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/87fc488b3f3e46c790a6e8d55b0364e8
I20260925 11:10:03.113695 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:03.091626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.8.181.1:34751)
0925 11:10:03.091872 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:03.091880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:03.091903 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:10:03.091972 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:03.091980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:03.092098 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:03.092214 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:03.094956 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.096397 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.099056 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.099970 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.100122 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.104446 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:03.104458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:03.104475 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:03.104525 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:03.108586 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.109661 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.109674 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.109680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.109863 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.110459 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.110505 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.110509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.111057 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:03.112728 (+ 1671us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.112891 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.113147 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 11:10:03.114250 23844 ts_tablet_manager.cc:1939] T 87fc488b3f3e46c790a6e8d55b0364e8 P a95f155cbfca486b92797d11bf397ca5: Deleting consensus metadata
I20260925 11:10:03.114799 23536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:03.120103 23845 ts_tablet_manager.cc:1918] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:03.120303 23845 log.cc:1196] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/87fc488b3f3e46c790a6e8d55b0364e8
I20260925 11:10:03.120857 23845 ts_tablet_manager.cc:1939] T 87fc488b3f3e46c790a6e8d55b0364e8 P c06d1b0593db44e690ccc8b6d567b3f7: Deleting consensus metadata
I20260925 11:10:03.122972 23340 catalog_manager.cc:4841] TS c06d1b0593db44e690ccc8b6d567b3f7 (127.8.181.2:34383): tablet 87fc488b3f3e46c790a6e8d55b0364e8 (table test-table [id=13d59b6327954c1b8bcc557983805052]) successfully deleted
I20260925 11:10:03.207556 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23393
I20260925 11:10:03.207728 23340 catalog_manager.cc:4841] TS a95f155cbfca486b92797d11bf397ca5 (127.8.181.1:34751): tablet 87fc488b3f3e46c790a6e8d55b0364e8 (table test-table [id=13d59b6327954c1b8bcc557983805052]) successfully deleted
I20260925 11:10:03.234146 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23545
I20260925 11:10:03.253937 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:03.235102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.8.181.2:34383)
0925 11:10:03.235306 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:03.235310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:03.235329 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:10:03.235511 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:03.235516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:03.235593 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:03.235679 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:03.237521 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.238462 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.239935 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.240344 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.240484 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.247445 (+ 6961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:03.247462 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:03.247477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:03.247520 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:03.250862 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.251728 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.251736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.251739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.251873 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.252273 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.252293 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.252295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.252701 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:03.253027 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.253363 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.253588 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:10:03.254886 23668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:03.347051 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23545
I20260925 11:10:03.365180 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23676
I20260925 11:10:03.384158 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:03.366490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.8.181.3:45243)
0925 11:10:03.366715 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:03.366721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:03.366772 (+ 51us) server_negotiation.cc:407] Connection header received
0925 11:10:03.366850 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:03.366855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:03.366957 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:03.367066 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:03.368503 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.369564 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.371760 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:03.372522 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:03.372713 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:03.375848 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:03.375871 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:03.375892 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:03.375949 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:03.380435 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.381489 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.381500 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.381503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.381722 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:03.382188 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:03.382194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:03.382197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:03.382750 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:03.383089 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:03.383455 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 11:10:03.383731 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:10:03.385305 23799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:03.512416 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23676
I20260925 11:10:03.532989 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23323
I20260925 11:10:03.534158 23386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:03.651212 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23323
I20260925 11:10:03.675767 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--webserver_interface=127.8.181.62
--webserver_port=44841
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:04.014979 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:04.015295 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:04.015448 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:04.023144 23850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:10:04.023275 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:04.023360 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:04.023447 23850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:10:04.023516 23850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:10:04.034140 23850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=44841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:04.035804 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:04.037595 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:04.055636 23855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:04.055696 23856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:04.056641 23858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:04.058270 23850 server_base.cc:1034] running on GCE node
I20260925 11:10:04.059795 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:04.061988 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:04.063374 23850 hybrid_clock.cc:648] HybridClock initialized: now 1790334604063316 us; error 58 us; skew 500 ppm
I20260925 11:10:04.063805 23850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:04 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:04.068207 23850 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:10:04.070528 23850 webserver.cc:460] Webserver started at http://127.8.181.62:44841/ using document root <none> and password file <none>
I20260925 11:10:04.071460 23850 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:04.071620 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:04.080047 23850 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 11:10:04.084713 23865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:04.086563 23850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260925 11:10:04.086800 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a4de74eae4a8417cbfed00bc2e21ba61"
format_stamp: "Formatted at 2026-09-25 11:09:57 on dist-test-slave-6042"
server_key: "18097937c1c3b85937d021604719dffd"
server_key_iv: "897696643e0821a9e15840b9cffbb5b9"
server_key_version: "encryptionkey@0"
I20260925 11:10:04.088246 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:04.145449 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:04.149660 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:04.149902 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:04.169749 23850 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:44335
I20260925 11:10:04.169767 23916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:44335 every 8 connection(s)
I20260925 11:10:04.171452 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:10:04.178843 23917 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:10:04.181721 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23850
I20260925 11:10:04.182369 23917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap starting.
I20260925 11:10:04.182621 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:34751
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=36299
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:10:04.203563 23917 log.cc:824] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:04.220110 23917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:10:04.220710 23917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61: Bootstrap complete.
I20260925 11:10:04.229936 23917 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:04.230965 23917 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Initialized, Role: FOLLOWER
I20260925 11:10:04.231571 23917 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:04.231833 23917 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:10:04.231964 23917 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:10:04.232138 23917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 2 FOLLOWER]: Advancing to term 3
I20260925 11:10:04.236368 23917 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:04.236905 23917 leader_election.cc:302] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4de74eae4a8417cbfed00bc2e21ba61; no voters:
I20260925 11:10:04.237607 23917 leader_election.cc:288] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 11:10:04.237751 23921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 11:10:04.239094 23921 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [term 3 LEADER]: Becoming Leader. State: Replica: a4de74eae4a8417cbfed00bc2e21ba61, State: Running, Role: LEADER
I20260925 11:10:04.239668 23921 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } }
I20260925 11:10:04.239984 23917 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:10:04.242413 23923 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4de74eae4a8417cbfed00bc2e21ba61. Latest consensus state: current_term: 3 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:10:04.242599 23922 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4de74eae4a8417cbfed00bc2e21ba61" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44335 } } }
I20260925 11:10:04.243007 23922 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:04.242898 23923 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4de74eae4a8417cbfed00bc2e21ba61 [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:04.252226 23928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:10:04.267298 23928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:10:04.268884 23928 catalog_manager.cc:1259] Loaded cluster ID: 40156d3ba3a54071af075d6462fa4341
I20260925 11:10:04.269028 23928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 11:10:04.271723 23928 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f795e7539ad google::LogMessage::Fail() at ??:0
@ 0x7f795e75688a google::LogMessage::SendToLog() at ??:0
@ 0x7f795e7534de google::LogMessage::Flush() at ??:0
@ 0x7f795e754a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f797418a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7974187976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f797420f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f797420f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f796139f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f79613a291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f79613a2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f796137deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f79646576db start_thread at ??:0
@ 0x7f795d5ed61f clone at ??:0
W20260925 11:10:04.598551 23919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:04.598860 23919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:04.599046 23919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:04.606668 23919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:04.606792 23919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:04.606967 23919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:10:04.617254 23919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:34751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=36299
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:04.618695 23919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:04.620071 23919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:04.634011 23958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:04.633962 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:04.634126 23961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:04.635043 23919 server_base.cc:1034] running on GCE node
I20260925 11:10:04.635710 23919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:04.636829 23919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:04.638056 23919 hybrid_clock.cc:648] HybridClock initialized: now 1790334604637980 us; error 76 us; skew 500 ppm
I20260925 11:10:04.638371 23919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:04 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:04.641661 23919 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:10:04.643316 23919 webserver.cc:460] Webserver started at http://127.8.181.1:36299/ using document root <none> and password file <none>
I20260925 11:10:04.644079 23919 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:04.644223 23919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:04.650132 23919 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 11:10:04.653169 23968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:04.654829 23919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:10:04.655021 23919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "a95f155cbfca486b92797d11bf397ca5"
format_stamp: "Formatted at 2026-09-25 11:09:58 on dist-test-slave-6042"
server_key: "13a001651154365df0658df18cf559e5"
server_key_iv: "a5e3d8d5b51179851323b5a93be4bfe0"
server_key_version: "encryptionkey@0"
I20260925 11:10:04.656100 23919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:04.675576 23919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:04.680006 23919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:04.680320 23919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:04.681334 23919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:04.682685 23919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:04.682816 23919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:04.682955 23919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:04.683032 23919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:04.715925 23919 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34751
I20260925 11:10:04.716013 24080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34751 every 8 connection(s)
I20260925 11:10:04.717373 23919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:10:04.718546 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 23919
I20260925 11:10:04.719257 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:34383
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=38023
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:10:04.722606 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:04.720595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:47725)
0925 11:10:04.721414 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:04.721669 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":346,"threads_started":1}
W20260925 11:10:04.723305 24081 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:44335 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:44335: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
I20260925 11:10:04.724100 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:04.723770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:36865)
0925 11:10:04.723936 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:04.724002 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 11:10:04.725124 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:04.724707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:41553)
0925 11:10:04.724845 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:04.724957 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 11:10:04.725401 24081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:10:05.049849 24084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:05.050145 24084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:05.050309 24084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:05.057806 24084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:05.057942 24084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:05.058125 24084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:10:05.068317 24084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:34383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=38023
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:05.069690 24084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:05.071099 24084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:05.084736 24089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:05.084686 24092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:05.084689 24090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:05.085664 24084 server_base.cc:1034] running on GCE node
I20260925 11:10:05.086267 24084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:05.087365 24084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:05.088598 24084 hybrid_clock.cc:648] HybridClock initialized: now 1790334605088549 us; error 43 us; skew 500 ppm
I20260925 11:10:05.088955 24084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:05 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:05.092218 24084 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:10:05.093729 24084 webserver.cc:460] Webserver started at http://127.8.181.2:38023/ using document root <none> and password file <none>
I20260925 11:10:05.094445 24084 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:05.094621 24084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:05.100242 24084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 11:10:05.103195 24099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.104526 24084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:10:05.104712 24084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "c06d1b0593db44e690ccc8b6d567b3f7"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "c7a6ddfe52111dd57cf63e94e496346e"
server_key_iv: "11567058c1d0e7b55e2bb049089d975a"
server_key_version: "encryptionkey@0"
I20260925 11:10:05.105695 24084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:05.134187 24084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:05.138882 24084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:05.139194 24084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:05.140277 24084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:05.141722 24084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:05.141870 24084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.141966 24084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:05.142023 24084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.177140 24084 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:34383
I20260925 11:10:05.177258 24211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:34383 every 8 connection(s)
I20260925 11:10:05.178716 24084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:10:05.183002 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24084
I20260925 11:10:05.183666 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:45243
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:44335
--webserver_port=34071
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:10:05.183985 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.181872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:51155)
0925 11:10:05.182700 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.182939 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":304,"threads_started":1}
W20260925 11:10:05.184603 24212 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:44335 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:44335: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
I20260925 11:10:05.185527 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.185207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:34637)
0925 11:10:05.185353 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.185421 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 11:10:05.186450 24214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.186130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:60197)
0925 11:10:05.186263 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.186343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 11:10:05.186800 24212 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:10:05.521575 24215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:05.521894 24215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:05.522050 24215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:05.529657 24215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:05.529783 24215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:05.529968 24215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:10:05.540274 24215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:45243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=34071
--tserver_master_addrs=127.8.181.62:44335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:05.541661 24215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:05.543043 24215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:05.556905 24223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:05.556998 24221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:05.556948 24220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:05.558004 24215 server_base.cc:1034] running on GCE node
I20260925 11:10:05.558754 24215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:05.559918 24215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:05.561148 24215 hybrid_clock.cc:648] HybridClock initialized: now 1790334605561094 us; error 46 us; skew 500 ppm
I20260925 11:10:05.561513 24215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:05 dist-test-slave-6042 krb5kdc[22770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:05.564927 24215 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:10:05.566619 24215 webserver.cc:460] Webserver started at http://127.8.181.3:34071/ using document root <none> and password file <none>
I20260925 11:10:05.567387 24215 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:05.567531 24215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:05.573335 24215 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:10:05.576776 24230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.578455 24215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:10:05.578712 24215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "3d3d18c81bca4b888b9fc10345a7c29f"
format_stamp: "Formatted at 2026-09-25 11:09:59 on dist-test-slave-6042"
server_key: "8d1aeafaf9e812a63e1234c1409379fd"
server_key_iv: "f2802d9e928450e200b7363d2ea9ad26"
server_key_version: "encryptionkey@0"
I20260925 11:10:05.579771 24215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:05.598208 24215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:05.603590 24215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:05.604085 24215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:05.605213 24215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:05.606796 24215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:05.606948 24215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.607079 24215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:05.607164 24215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:05.644804 24215 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45243
I20260925 11:10:05.645051 24342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45243 every 8 connection(s)
I20260925 11:10:05.646658 24215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:10:05.649302 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24215
I20260925 11:10:05.652657 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.650393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:54569)
0925 11:10:05.651476 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.651698 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":443,"threads_started":1}
W20260925 11:10:05.653635 24343 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:44335 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:44335: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
I20260925 11:10:05.654856 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.654318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:47945)
0925 11:10:05.654573 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.654652 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 11:10:05.655944 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.655513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:46277)
0925 11:10:05.655695 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.655819 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 11:10:05.656209 24343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 11:10:05.726902 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:05.726029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:51549)
0925 11:10:05.726558 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:05.726670 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260925 11:10:06.188170 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:06.187329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:33817)
0925 11:10:06.187793 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:06.187886 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260925 11:10:06.657922 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:06.656989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:39499)
0925 11:10:06.657532 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:06.657675 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":278,"threads_started":1}
I20260925 11:10:06.728621 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:06.727869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:48487)
0925 11:10:06.728329 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:06.728414 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260925 11:10:07.189985 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:07.189181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:48165)
0925 11:10:07.189699 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:07.189787 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
I20260925 11:10:07.660061 24352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:07.659030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:48229)
0925 11:10:07.659581 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:07.659747 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":312,"threads_started":1}
I20260925 11:10:07.730357 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:07.729594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:51733)
0925 11:10:07.730098 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:07.730189 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":325,"threads_started":1}
I20260925 11:10:08.191804 24354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:08.191054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:60589)
0925 11:10:08.191535 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:08.191624 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260925 11:10:08.661770 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:08.661085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:38079)
0925 11:10:08.661518 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:08.661597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260925 11:10:08.732183 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:08.731421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:38685)
0925 11:10:08.731894 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:08.731982 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260925 11:10:09.193534 24357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:09.192811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:60027)
0925 11:10:09.193248 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:09.193335 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":211,"threads_started":1}
I20260925 11:10:09.663676 24358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:09.662946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:59957)
0925 11:10:09.663417 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:09.663505 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260925 11:10:09.734107 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:09.733270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:45103)
0925 11:10:09.733772 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:09.733864 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260925 11:10:10.195344 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:10.194537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:56371)
0925 11:10:10.195070 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:10.195160 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":300,"threads_started":1}
I20260925 11:10:10.665390 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:10.664656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:43129)
0925 11:10:10.665127 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:10.665220 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260925 11:10:10.735919 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:10.735054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:46583)
0925 11:10:10.735574 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:10.735669 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":280,"threads_started":1}
I20260925 11:10:11.197111 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:11.196361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:54047)
0925 11:10:11.196859 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:11.196947 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":248,"threads_started":1}
I20260925 11:10:11.667410 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:11.666647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:41117)
0925 11:10:11.667131 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:11.667236 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260925 11:10:11.737866 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:11.736922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:49093)
0925 11:10:11.737443 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:11.737584 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":249,"threads_started":1}
I20260925 11:10:12.199066 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:12.198161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:43107)
0925 11:10:12.198753 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:12.198852 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":297,"threads_started":1}
I20260925 11:10:12.669240 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:12.668450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:48333)
0925 11:10:12.668943 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:12.669032 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":216,"threads_started":1}
I20260925 11:10:12.739631 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:12.738912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:48591)
0925 11:10:12.739345 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:12.739433 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260925 11:10:13.201071 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:13.200228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:58011)
0925 11:10:13.200731 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:13.200819 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
I20260925 11:10:13.671090 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:13.670221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:43513)
0925 11:10:13.670780 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:13.670867 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":316,"threads_started":1}
I20260925 11:10:13.741402 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:13.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:50327)
0925 11:10:13.741125 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:13.741216 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
I20260925 11:10:14.203390 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:14.202240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:39147)
0925 11:10:14.203009 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:14.203117 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":405,"threads_started":1}
I20260925 11:10:14.672830 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:14.672116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:60913)
0925 11:10:14.672571 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:14.672654 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260925 11:10:14.743249 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:14.742451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:50315)
0925 11:10:14.742992 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:14.743084 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260925 11:10:15.205509 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:15.204561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:33255)
0925 11:10:15.205230 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:15.205346 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":425,"threads_started":1}
I20260925 11:10:15.675282 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:15.674117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:36487)
0925 11:10:15.674847 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:15.674984 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":444,"threads_started":1}
I20260925 11:10:15.745092 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:15.744284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:50277)
0925 11:10:15.744762 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:15.744909 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260925 11:10:16.208717 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:16.206538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:60363)
0925 11:10:16.208377 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:16.208488 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1735,"thread_start_us":1628,"threads_started":1}
I20260925 11:10:16.677251 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:16.676457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:51675)
0925 11:10:16.676979 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:16.677071 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":243,"threads_started":1}
I20260925 11:10:16.746997 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:16.746103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:46823)
0925 11:10:16.746670 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:16.746833 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
I20260925 11:10:17.210726 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:17.209869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:41085)
0925 11:10:17.210314 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:17.210429 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260925 11:10:17.679008 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:17.678243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:33123)
0925 11:10:17.678726 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:17.678806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":289,"threads_started":1}
I20260925 11:10:17.748725 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:17.747968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:48675)
0925 11:10:17.748442 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:17.748539 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260925 11:10:18.212558 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:18.211762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:41387)
0925 11:10:18.212264 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:18.212362 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":257,"threads_started":1}
I20260925 11:10:18.680824 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:18.680041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:45371)
0925 11:10:18.680529 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:18.680631 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":227,"threads_started":1}
I20260925 11:10:18.750736 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:18.749748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:51467)
0925 11:10:18.750335 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:18.750436 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":338,"threads_started":1}
I20260925 11:10:19.214627 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:19.213668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:58759)
0925 11:10:19.214200 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:19.214303 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
I20260925 11:10:19.682538 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:19.681750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:39351)
0925 11:10:19.682198 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:19.682281 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":240,"threads_started":1}
I20260925 11:10:19.752697 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:19.751928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:58191)
0925 11:10:19.752361 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:19.752506 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260925 11:10:20.216516 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.215739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.2:58541)
0925 11:10:20.216196 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:20.216309 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260925 11:10:20.660395 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 23919
I20260925 11:10:20.684708 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.683813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.3:41065)
0925 11:10:20.684353 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:20.684487 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260925 11:10:20.692523 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.661702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.8.181.1:34751)
0925 11:10:20.662546 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:20.662570 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:20.662624 (+ 54us) server_negotiation.cc:407] Connection header received
0925 11:10:20.664688 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:20.664699 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:20.665327 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:20.665664 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:20.667367 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.668672 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.670031 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.670865 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.671010 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:20.684745 (+ 13735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:20.684759 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:20.684787 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:20.684874 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:20.688833 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.689675 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.689683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.689691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.689960 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.690397 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.690401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.690403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.691140 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:20.691374 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:20.691858 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 11:10:20.692237 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":417,"threads_started":1}
I20260925 11:10:20.694100 24076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:20.780310 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.778583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44335 (local address 127.8.181.1:55891)
0925 11:10:20.779749 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:20.779967 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:44335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1027,"thread_start_us":875,"threads_started":1}
I20260925 11:10:20.782650 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 23919
I20260925 11:10:20.800678 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24084
I20260925 11:10:20.819015 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.801668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.8.181.2:34383)
0925 11:10:20.802164 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:20.802178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:20.802214 (+ 36us) server_negotiation.cc:407] Connection header received
0925 11:10:20.803721 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:20.803730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:20.804158 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:20.804453 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:20.805807 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.807236 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.808527 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.809207 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.809342 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:20.812235 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:20.812243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:20.812260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:20.812302 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:20.815749 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.816513 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.816519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.816527 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.816697 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.817054 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.817057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.817059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.817601 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:20.817819 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:20.818229 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 11:10:20.818702 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260925 11:10:20.820472 24207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:20.904731 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24084
I20260925 11:10:20.922557 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24215
I20260925 11:10:20.940802 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:20.923631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.8.181.3:45243)
0925 11:10:20.924291 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:20.924304 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:20.924343 (+ 39us) server_negotiation.cc:407] Connection header received
0925 11:10:20.925976 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:20.925987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:20.926414 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:20.926902 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:20.928228 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.929441 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.930883 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:20.931575 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:20.931715 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:20.934303 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:20.934311 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:20.934327 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:20.934375 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:20.937387 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.938216 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.938226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.938238 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.938404 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:20.938957 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:20.938960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:20.938962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:20.939513 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:20.939741 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:20.940151 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 11:10:20.940498 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":364,"threads_started":1}
I20260925 11:10:20.942274 24338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:21.027004 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24215
2026-09-25T11:10:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24919 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:10:21 dist-test-slave-6042 krb5kdc[24401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:10:21 dist-test-slave-6042 krb5kdc[24401](info): set up 2 sockets
Sep 25 11:10:21 dist-test-slave-6042 krb5kdc[24401](info): commencing operation
krb5kdc: starting...
W20260925 11:10:22.135406 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:10:22 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:10:22.391235 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:22.735046 24414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:22.735348 24414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:22.735538 24414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:22.743465 24414 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 11:10:22.743583 24414 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 11:10:22.743672 24414 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 11:10:22.743767 24414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:10:22.743861 24414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:22.743983 24414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:22.744043 24414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:10:22.744091 24414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:10:22.754344 24414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:22.755780 24414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:22.757269 24414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:22.770277 24419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:22.770310 24420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:22.770352 24422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:22.771435 24414 server_base.cc:1034] running on GCE node
I20260925 11:10:22.772279 24414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:22.773808 24414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:22.775053 24414 hybrid_clock.cc:648] HybridClock initialized: now 1790334622774985 us; error 47 us; skew 500 ppm
I20260925 11:10:22.775399 24414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:22 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:22.778573 24414 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:10:22.780161 24414 webserver.cc:460] Webserver started at http://127.8.181.62:32847/ using document root <none> and password file <none>
I20260925 11:10:22.780874 24414 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:22.781008 24414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:22.781323 24414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:10:22.784405 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "d6a89672f58b40b0b85dde258173a8ee"
format_stamp: "Formatted at 2026-09-25 11:10:22 on dist-test-slave-6042"
server_key: "9d9ed6fa461ae22221a1bd56d1de9a9d"
server_key_iv: "3aeb7bda4639555c39be43f6c516aa70"
server_key_version: "encryptionkey@0"
I20260925 11:10:22.785202 24414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "d6a89672f58b40b0b85dde258173a8ee"
format_stamp: "Formatted at 2026-09-25 11:10:22 on dist-test-slave-6042"
server_key: "9d9ed6fa461ae22221a1bd56d1de9a9d"
server_key_iv: "3aeb7bda4639555c39be43f6c516aa70"
server_key_version: "encryptionkey@0"
I20260925 11:10:22.790377 24414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 11:10:22.793929 24429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:22.795311 24414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:10:22.795509 24414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d6a89672f58b40b0b85dde258173a8ee"
format_stamp: "Formatted at 2026-09-25 11:10:22 on dist-test-slave-6042"
server_key: "9d9ed6fa461ae22221a1bd56d1de9a9d"
server_key_iv: "3aeb7bda4639555c39be43f6c516aa70"
server_key_version: "encryptionkey@0"
I20260925 11:10:22.795766 24414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:22.813202 24414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:22.817421 24414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:22.817706 24414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:22.836748 24414 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:36663
I20260925 11:10:22.836768 24480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:36663 every 8 connection(s)
I20260925 11:10:22.838140 24414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:10:22.841879 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:22.846499 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24414
I20260925 11:10:22.846732 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:10:22.847580 8916 external_mini_cluster.cc:1428] Setting key b7b4fcd06c30c8080b8b977cfbf4b0b7
I20260925 11:10:22.854432 24481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap starting.
I20260925 11:10:22.858605 24481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:22.859833 24481 log.cc:824] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:22.863529 24481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: No bootstrap required, opened a new log
Sep 25 11:10:22 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:22.874150 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:22.849785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.8.181.62:36663)
0925 11:10:22.850609 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:22.850621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:22.850671 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:10:22.852710 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:22.852735 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:22.853150 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:22.853622 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:22.855539 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:22.857087 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:22.859110 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:22.859812 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:22.860005 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:22.864957 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:22.864970 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:22.864990 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:22.865034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:22.869189 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:22.870037 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:22.870045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:22.870055 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:22.870271 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:22.870694 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:22.870700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:22.870703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:22.871509 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:22.871817 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:22.872182 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 11:10:22.872680 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":371,"threads_started":1}
I20260925 11:10:22.874812 24481 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:22.875226 24481 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:22.875344 24481 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Initialized, Role: FOLLOWER
I20260925 11:10:22.875948 24481 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:22.876204 24481 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:10:22.876360 24481 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:10:22.876557 24481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:22.878993 24481 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:22.879496 24481 leader_election.cc:302] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a89672f58b40b0b85dde258173a8ee; no voters:
I20260925 11:10:22.880108 24481 leader_election.cc:288] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:10:22.880236 24486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:10:22.881006 24486 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 LEADER]: Becoming Leader. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Running, Role: LEADER
I20260925 11:10:22.881572 24486 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:22.882302 24481 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:10:22.884835 24487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:10:22.885934 24487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:22.885527 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a89672f58b40b0b85dde258173a8ee. Latest consensus state: current_term: 1 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:10:22.886253 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:22.887351 24494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:10:22.893062 24494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:10:22.902446 24494 catalog_manager.cc:1347] Generated new cluster ID: 48feb9b1252a42f08b052d9afc799109
I20260925 11:10:22.902645 24494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:10:22.914053 24494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:10:22.916092 24494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:10:22.926049 24494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Generated new TSK 0
I20260925 11:10:22.927227 24494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:10:22.990882 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:23.334905 24507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:23.335201 24507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:23.335359 24507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:23.342763 24507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:23.342892 24507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:23.343067 24507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:10:23.353204 24507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:23.354636 24507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:23.356000 24507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:23.370404 24512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:23.370337 24515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:23.370378 24513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:23.371096 24507 server_base.cc:1034] running on GCE node
I20260925 11:10:23.371912 24507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:23.372993 24507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:23.374264 24507 hybrid_clock.cc:648] HybridClock initialized: now 1790334623374212 us; error 63 us; skew 500 ppm
I20260925 11:10:23.374697 24507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:23 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:23.378240 24507 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:10:23.379881 24507 webserver.cc:460] Webserver started at http://127.8.181.1:45739/ using document root <none> and password file <none>
I20260925 11:10:23.380630 24507 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:23.380790 24507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:23.381131 24507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:10:23.384368 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "106bcf16ce3946a999d73b26187ad37a"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "ed13808c88eb98c3948ebb46650d06a2"
server_key_iv: "75c639a37dbe98b66c326ab8d3efaf92"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.385210 24507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "106bcf16ce3946a999d73b26187ad37a"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "ed13808c88eb98c3948ebb46650d06a2"
server_key_iv: "75c639a37dbe98b66c326ab8d3efaf92"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.391083 24507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 11:10:23.394784 24522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.396106 24507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:10:23.396314 24507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "106bcf16ce3946a999d73b26187ad37a"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "ed13808c88eb98c3948ebb46650d06a2"
server_key_iv: "75c639a37dbe98b66c326ab8d3efaf92"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.396531 24507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:23.413061 24507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:23.417168 24507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:23.417436 24507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:23.418365 24507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:23.419812 24507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:23.419937 24507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.420081 24507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:23.420163 24507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.456480 24507 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33589
I20260925 11:10:23.456614 24634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33589 every 8 connection(s)
I20260925 11:10:23.458007 24507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:10:23.466163 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24507
I20260925 11:10:23.466418 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:10:23.467288 8916 external_mini_cluster.cc:1428] Setting key c739aaa6a2c1b2e9bea4916c4f272c88
Sep 25 11:10:23 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:23.481248 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:23.460723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:51191 (local address 127.8.181.62:36663)
0925 11:10:23.461210 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:23.461217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:23.462200 (+ 983us) server_negotiation.cc:407] Connection header received
0925 11:10:23.464047 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:23.464052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:23.464122 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:23.464223 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:23.465946 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:23.467200 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:23.469307 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:23.469868 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:23.470066 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:23.474031 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:23.474041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:23.474049 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:23.474097 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:23.477735 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:23.478841 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:23.478845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:23.478848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:23.478980 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:23.479491 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:23.479494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:23.479496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:23.479757 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:23.479868 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:23.480797 (+ 929us) server_negotiation.cc:299] Negotiation successful
0925 11:10:23.480983 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260925 11:10:23.482921 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:23.461336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:51191)
0925 11:10:23.462035 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:23.462080 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:23.463786 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:23.464443 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:23.464452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:23.464986 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:23.465636 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:23.465655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:23.467423 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:23.467427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:23.469137 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:23.469174 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:23.470139 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:23.470143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:23.470343 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:23.471217 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:23.471249 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:23.473751 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:23.477909 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:23.477921 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:23.477945 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:23.478700 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:23.479096 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:23.479104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:23.479110 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:23.479362 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:23.479884 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:23.479895 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:23.480380 (+ 485us) client_negotiation.cc:769] Sending connection context
0925 11:10:23.480707 (+ 327us) client_negotiation.cc:240] Negotiation successful
0925 11:10:23.481306 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":283,"threads_started":1}
I20260925 11:10:23.484731 24635 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:23.485061 24635 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:23.485828 24635 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:23.488379 24446 ts_manager.cc:194] Registered new tserver with Master: 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589)
I20260925 11:10:23.490742 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:51191
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:10:23.526350 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:23.861235 24641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:23.861548 24641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:23.861721 24641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:23.869370 24641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:23.869491 24641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:23.869721 24641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:10:23.879979 24641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:23.881309 24641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:23.882716 24641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:23.896811 24646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:23.896736 24649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:23.896768 24647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:23.897662 24641 server_base.cc:1034] running on GCE node
I20260925 11:10:23.898239 24641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:23.899223 24641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:23.900429 24641 hybrid_clock.cc:648] HybridClock initialized: now 1790334623900363 us; error 45 us; skew 500 ppm
I20260925 11:10:23.900704 24641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:23 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:23.904297 24641 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:10:23.905850 24641 webserver.cc:460] Webserver started at http://127.8.181.2:41733/ using document root <none> and password file <none>
I20260925 11:10:23.906657 24641 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:23.906804 24641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:23.907137 24641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:10:23.910236 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "259be75582e74ec582a119144f4e42c0"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "eca5b99d37191d40f2cb939f2f9832d7"
server_key_iv: "6b32821f1ffd18653e9a44d0e93d0d24"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.911125 24641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "259be75582e74ec582a119144f4e42c0"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "eca5b99d37191d40f2cb939f2f9832d7"
server_key_iv: "6b32821f1ffd18653e9a44d0e93d0d24"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.916688 24641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 11:10:23.920528 24656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.922050 24641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:10:23.922240 24641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "259be75582e74ec582a119144f4e42c0"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "eca5b99d37191d40f2cb939f2f9832d7"
server_key_iv: "6b32821f1ffd18653e9a44d0e93d0d24"
server_key_version: "encryptionkey@0"
I20260925 11:10:23.922506 24641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:23.953339 24641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:23.957516 24641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:23.957823 24641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:23.958971 24641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:23.960469 24641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:23.960541 24641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.960669 24641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:23.960729 24641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:23.995010 24641 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:46443
I20260925 11:10:23.995126 24768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:46443 every 8 connection(s)
I20260925 11:10:23.996407 24641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:10:24.000797 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24641
I20260925 11:10:24.001037 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:10:24.001466 8916 external_mini_cluster.cc:1428] Setting key c68f93b71d33376ad8e1b9b505b218fd
Sep 25 11:10:24 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:24.017761 24771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:23.999069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:59269 (local address 127.8.181.62:36663)
0925 11:10:23.999530 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:23.999534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.000664 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 11:10:24.003194 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.003200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.003284 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.003414 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:24.004988 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.005715 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.007372 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.007820 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.007970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.011663 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:24.011668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:24.011671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:24.011705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:24.014410 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.015290 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.015301 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.015305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.015450 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.015916 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.015928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.015932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.016209 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:24.016367 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.017098 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.017335 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
I20260925 11:10:24.018891 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:23.999646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:59269)
0925 11:10:24.000443 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.000502 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.002858 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.003670 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.003678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.004216 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:24.004805 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.004822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.005866 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.005872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.007220 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.007232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.008065 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.008072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.008332 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.009052 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:24.009074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:24.011403 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:24.014623 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:24.014632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:24.014652 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:24.015150 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:24.015568 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:24.015573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:24.015577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:24.015790 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:24.016341 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:24.016349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:24.016720 (+ 371us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.016997 (+ 277us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.017564 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":275,"threads_started":1}
I20260925 11:10:24.020678 24769 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:24.021059 24769 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:24.022049 24769 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:24.023801 24446 ts_manager.cc:194] Registered new tserver with Master: 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443)
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
I20260925 11:10:24.024722 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:59269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:10:24.058387 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:24.414047 24775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:24.414343 24775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:24.414438 24775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:24.422322 24775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:24.422441 24775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:24.422654 24775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:10:24.432852 24775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:24.434156 24775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:24.435474 24775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:24.449153 24780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:24.449098 24781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:24.449311 24783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:24.450241 24775 server_base.cc:1034] running on GCE node
I20260925 11:10:24.451112 24775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:24.452217 24775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:24.453467 24775 hybrid_clock.cc:648] HybridClock initialized: now 1790334624453411 us; error 75 us; skew 500 ppm
I20260925 11:10:24.453823 24775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:24 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:24.457391 24775 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:10:24.461378 24775 webserver.cc:460] Webserver started at http://127.8.181.3:33431/ using document root <none> and password file <none>
I20260925 11:10:24.462105 24775 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:24.462262 24775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:24.462649 24775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:10:24.465739 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "96af50b917a74c37b42de23cc8babb0e"
format_stamp: "Formatted at 2026-09-25 11:10:24 on dist-test-slave-6042"
server_key: "0fecb597ea5bda4869d80fee27945bd3"
server_key_iv: "529c5fa0b6d0f68127ea999e379edc86"
server_key_version: "encryptionkey@0"
I20260925 11:10:24.466603 24775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "96af50b917a74c37b42de23cc8babb0e"
format_stamp: "Formatted at 2026-09-25 11:10:24 on dist-test-slave-6042"
server_key: "0fecb597ea5bda4869d80fee27945bd3"
server_key_iv: "529c5fa0b6d0f68127ea999e379edc86"
server_key_version: "encryptionkey@0"
I20260925 11:10:24.471851 24775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 11:10:24.475315 24790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:24.476652 24775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:10:24.476831 24775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "96af50b917a74c37b42de23cc8babb0e"
format_stamp: "Formatted at 2026-09-25 11:10:24 on dist-test-slave-6042"
server_key: "0fecb597ea5bda4869d80fee27945bd3"
server_key_iv: "529c5fa0b6d0f68127ea999e379edc86"
server_key_version: "encryptionkey@0"
I20260925 11:10:24.477051 24775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:24.494136 24635 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:24.520766 24775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:24.525972 24775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:24.526373 24775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:24.527835 24775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:24.529686 24775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:24.529791 24775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:24.529925 24775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:24.529989 24775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:24.564260 24775 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:44023
I20260925 11:10:24.564332 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:44023 every 8 connection(s)
I20260925 11:10:24.565555 24775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:10:24.574148 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24775
I20260925 11:10:24.574369 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:10:24.574848 8916 external_mini_cluster.cc:1428] Setting key 25c69fbdc071f06243f225c40dbe71f9
Sep 25 11:10:24 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:24.586369 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.568156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55925 (local address 127.8.181.62:36663)
0925 11:10:24.568604 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.568610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.569580 (+ 970us) server_negotiation.cc:407] Connection header received
0925 11:10:24.571505 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.571510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.571584 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.571687 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:24.573356 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.574124 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.576037 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.576614 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.576753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.580425 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:24.580431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:24.580434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:24.580466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:24.583466 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.584304 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.584308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.584310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.584434 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.584822 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.584825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.584827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.585127 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:24.585244 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.585911 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.586080 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":194,"threads_started":1}
I20260925 11:10:24.587556 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.568744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:55925)
0925 11:10:24.569413 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.569457 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.571226 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.571889 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.571931 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.572464 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:24.573143 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.573169 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.574306 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.574314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.575876 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.575889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.576876 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.576891 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.577203 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.577946 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:24.577980 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:24.580231 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:24.583614 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:24.583621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:24.583636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:24.584191 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:24.584530 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:24.584533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:24.584535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:24.584717 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:24.585235 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:24.585241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:24.585565 (+ 324us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.585797 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.586210 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":276,"threads_started":1}
I20260925 11:10:24.589218 24903 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:24.589566 24903 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:24.590524 24903 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:24.591969 24446 ts_manager.cc:194] Registered new tserver with Master: 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
I20260925 11:10:24.592785 24446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:55925
I20260925 11:10:24.603402 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:10:24.622378 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.608284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.8.181.62:36663)
0925 11:10:24.608531 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.608538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.608992 (+ 454us) server_negotiation.cc:407] Connection header received
0925 11:10:24.609219 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.609225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.609312 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.609431 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:24.611023 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.611772 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.613221 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.613670 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.613818 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.616702 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:24.616710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:24.616720 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:24.616756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:24.619490 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.620273 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.620279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.620282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.620440 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.620833 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.620841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.620844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.621193 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:24.621466 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.621787 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.622040 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 11:10:24.627120 24446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:10:24.634177 24446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:10:24.659775 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.647697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46443 (local address 127.0.0.1:41186)
0925 11:10:24.650218 (+ 2521us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.650244 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.650434 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.651968 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.651975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.652030 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:24.652637 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.652652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654965 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.654972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.657328 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.657338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.658658 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.658664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.658860 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.658924 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.659070 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.659281 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2357,"mutex_wait_us":30,"spinlock_wait_cycles":7168,"thread_start_us":1265,"threads_started":1}
I20260925 11:10:24.660393 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.649558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.8.181.1:33589)
0925 11:10:24.650198 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.650210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.651159 (+ 949us) server_negotiation.cc:407] Connection header received
0925 11:10:24.651511 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.651520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.651744 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.651905 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:24.653116 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654144 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.657258 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.658774 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.658913 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.659133 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.659890 (+ 757us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.660064 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":376,"threads_started":1}
I20260925 11:10:24.660575 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.647736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.8.181.3:44023)
0925 11:10:24.651402 (+ 3666us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.651410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.651442 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:10:24.651535 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.651542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.651764 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.652374 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:24.653032 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654634 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.657176 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.658767 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.658946 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.659143 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.660012 (+ 869us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.660179 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3485,"thread_start_us":3315,"threads_started":1}
I20260925 11:10:24.659891 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.647418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44023 (local address 127.0.0.1:42076)
0925 11:10:24.648485 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.648541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.648726 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.651971 (+ 3245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.651977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.652030 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:24.652764 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.652784 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654832 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.654838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.656991 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.657007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.659009 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.659018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.659199 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.659233 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.659316 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.659479 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":213,"spinlock_wait_cycles":1792,"thread_start_us":2492,"threads_started":1}
I20260925 11:10:24.660980 24919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.648450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:37812)
0925 11:10:24.651036 (+ 2586us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.651059 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.651265 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.652234 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.652241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.652275 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:24.652894 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.652908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654293 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.654299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.657043 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.657058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.659291 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.659299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.659455 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.659494 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.659589 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.659733 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"mutex_wait_us":97,"thread_start_us":2259,"threads_started":1}
I20260925 11:10:24.661010 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.648931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.8.181.2:46443)
0925 11:10:24.649544 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.649552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.650383 (+ 831us) server_negotiation.cc:407] Connection header received
0925 11:10:24.650758 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.650768 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.651465 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.651665 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:24.652836 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.654566 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.657485 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.658434 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.658612 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.658827 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.660478 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.660659 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":814,"threads_started":1}
I20260925 11:10:24.664539 24704 tablet_service.cc:1467] Processing CreateTablet for tablet c38b898622d34e228f5cbd7c18743fe8 (DEFAULT_TABLE table=test-table [id=4b5c0972a3de4134a72e1ead69d555d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:24.664671 24838 tablet_service.cc:1467] Processing CreateTablet for tablet c38b898622d34e228f5cbd7c18743fe8 (DEFAULT_TABLE table=test-table [id=4b5c0972a3de4134a72e1ead69d555d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:24.664671 24570 tablet_service.cc:1467] Processing CreateTablet for tablet c38b898622d34e228f5cbd7c18743fe8 (DEFAULT_TABLE table=test-table [id=4b5c0972a3de4134a72e1ead69d555d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:24.665977 24704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38b898622d34e228f5cbd7c18743fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:24.666062 24570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38b898622d34e228f5cbd7c18743fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:24.666427 24838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38b898622d34e228f5cbd7c18743fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:24.677588 24922 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Bootstrap starting.
I20260925 11:10:24.680940 24923 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Bootstrap starting.
I20260925 11:10:24.682449 24922 tablet_bootstrap.cc:654] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:24.682765 24924 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Bootstrap starting.
I20260925 11:10:24.684051 24922 log.cc:824] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:24.684737 24923 tablet_bootstrap.cc:654] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:24.685825 24923 log.cc:824] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:24.686046 24924 tablet_bootstrap.cc:654] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:24.687399 24924 log.cc:824] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:24.687520 24922 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: No bootstrap required, opened a new log
I20260925 11:10:24.687794 24922 ts_tablet_manager.cc:1397] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 11:10:24.689623 24923 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: No bootstrap required, opened a new log
I20260925 11:10:24.689898 24923 ts_tablet_manager.cc:1397] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 11:10:24.693053 24924 tablet_bootstrap.cc:492] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: No bootstrap required, opened a new log
I20260925 11:10:24.693378 24924 ts_tablet_manager.cc:1397] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260925 11:10:24.698907 24922 raft_consensus.cc:348] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.699432 24922 raft_consensus.cc:374] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:24.699532 24922 raft_consensus.cc:729] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259be75582e74ec582a119144f4e42c0, State: Initialized, Role: FOLLOWER
I20260925 11:10:24.700332 24922 consensus_queue.cc:260] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.701534 24769 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:24.701526 24924 raft_consensus.cc:348] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.701560 24923 raft_consensus.cc:348] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.701957 24924 raft_consensus.cc:374] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:24.701972 24923 raft_consensus.cc:374] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:24.702062 24924 raft_consensus.cc:729] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106bcf16ce3946a999d73b26187ad37a, State: Initialized, Role: FOLLOWER
I20260925 11:10:24.702059 24923 raft_consensus.cc:729] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96af50b917a74c37b42de23cc8babb0e, State: Initialized, Role: FOLLOWER
I20260925 11:10:24.702668 24923 consensus_queue.cc:260] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.702765 24924 consensus_queue.cc:260] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.703199 24922 ts_tablet_manager.cc:1428] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 11:10:24.704640 24903 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:24.704859 24924 ts_tablet_manager.cc:1428] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260925 11:10:24.706010 24923 ts_tablet_manager.cc:1428] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260925 11:10:24.710539 24636 tablet.cc:2369] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:10:24.724514 24929 raft_consensus.cc:482] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:10:24.724835 24929 raft_consensus.cc:504] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.726770 24929 leader_election.cc:288] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589)
I20260925 11:10:24.733484 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.727491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:46935 (local address 127.8.181.2:46443)
0925 11:10:24.727695 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.727699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.727715 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:10:24.727775 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.727779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.727888 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.727986 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:24.728615 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.729619 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.731559 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.732653 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.732755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.732836 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.733194 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.733290 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:10:24.733633 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.726995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46443 (local address 127.8.181.3:46935)
0925 11:10:24.727226 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.727248 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.727388 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.728173 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.728177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.728247 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:24.728448 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.728461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.729786 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.729792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.731380 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.731392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.732814 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.732820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.733016 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.733058 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.733160 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.733307 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"spinlock_wait_cycles":17152}
I20260925 11:10:24.734344 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.727305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:60183 (local address 127.8.181.1:33589)
0925 11:10:24.727547 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.727555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.728339 (+ 784us) server_negotiation.cc:407] Connection header received
0925 11:10:24.728428 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.728431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.728489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.728590 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:24.729355 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.730316 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.732375 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.733450 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.733583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.733691 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.734034 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.734131 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:10:24.734584 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.727289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.8.181.3:60183)
0925 11:10:24.728037 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:24.728051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:24.728203 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:24.728759 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:24.728764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:24.728788 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:24.729181 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.729192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.730525 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.730531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.732216 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:24.732230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.733674 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.733682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:24.733859 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.733891 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 11:10:24.733983 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 11:10:24.734123 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":443,"threads_started":1}
I20260925 11:10:24.734902 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38b898622d34e228f5cbd7c18743fe8" candidate_uuid: "96af50b917a74c37b42de23cc8babb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259be75582e74ec582a119144f4e42c0" is_pre_election: true
I20260925 11:10:24.735453 24724 raft_consensus.cc:2454] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96af50b917a74c37b42de23cc8babb0e in term 0.
I20260925 11:10:24.735633 24590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38b898622d34e228f5cbd7c18743fe8" candidate_uuid: "96af50b917a74c37b42de23cc8babb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106bcf16ce3946a999d73b26187ad37a" is_pre_election: true
I20260925 11:10:24.736148 24590 raft_consensus.cc:2454] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96af50b917a74c37b42de23cc8babb0e in term 0.
I20260925 11:10:24.736207 24792 leader_election.cc:302] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e; no voters:
I20260925 11:10:24.736686 24929 raft_consensus.cc:2781] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:10:24.736833 24929 raft_consensus.cc:482] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:10:24.736953 24929 raft_consensus.cc:3037] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:24.739197 24929 raft_consensus.cc:504] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
I20260925 11:10:24.739804 24929 leader_election.cc:288] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [CANDIDATE]: Term 1 election: Requested vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589)
I20260925 11:10:24.740269 24590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38b898622d34e228f5cbd7c18743fe8" candidate_uuid: "96af50b917a74c37b42de23cc8babb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106bcf16ce3946a999d73b26187ad37a"
I20260925 11:10:24.740269 24724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38b898622d34e228f5cbd7c18743fe8" candidate_uuid: "96af50b917a74c37b42de23cc8babb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259be75582e74ec582a119144f4e42c0"
I20260925 11:10:24.740483 24724 raft_consensus.cc:3037] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:24.740483 24590 raft_consensus.cc:3037] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:24.743779 24590 raft_consensus.cc:2454] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96af50b917a74c37b42de23cc8babb0e in term 1.
I20260925 11:10:24.743780 24724 raft_consensus.cc:2454] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96af50b917a74c37b42de23cc8babb0e in term 1.
I20260925 11:10:24.744256 24792 leader_election.cc:302] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e; no voters:
I20260925 11:10:24.744612 24929 raft_consensus.cc:2781] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:10:24.745131 24929 raft_consensus.cc:686] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 1 LEADER]: Becoming Leader. State: Replica: 96af50b917a74c37b42de23cc8babb0e, State: Running, Role: LEADER
I20260925 11:10:24.745781 24929 consensus_queue.cc:237] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } }
W20260925 11:10:24.748562 24770 tablet.cc:2369] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:10:24.751214 24445 catalog_manager.cc:5495] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 96af50b917a74c37b42de23cc8babb0e (127.8.181.3). New cstate: current_term: 1 leader_uuid: "96af50b917a74c37b42de23cc8babb0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } health_report { overall_health: UNKNOWN } } }
I20260925 11:10:24.786979 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.781052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.8.181.3:44023)
0925 11:10:24.781286 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.781291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.781312 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:10:24.781555 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.781558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.781647 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.781827 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:24.782253 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.783057 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.784697 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.785145 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.785290 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.785820 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:24.786332 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:10:24.786529 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.786580 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.786664 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:10:24.795614 24590 raft_consensus.cc:1261] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Refusing update from remote peer 96af50b917a74c37b42de23cc8babb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:24.795614 24724 raft_consensus.cc:1261] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Refusing update from remote peer 96af50b917a74c37b42de23cc8babb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:24.796823 24929 consensus_queue.cc:1035] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:24.797204 24932 consensus_queue.cc:1035] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:24.818415 24937 mvcc.cc:205] Tried to move back new op lower bound from 7333210623151009792 to 7333210622963806208. Current Snapshot: MvccSnapshot[applied={T|T < 7333210623151009792}]
I20260925 11:10:24.826618 24446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:24.826963 24446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:24.830503 24446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Sending DeleteTablet for 3 replicas of tablet c38b898622d34e228f5cbd7c18743fe8
I20260925 11:10:24.831749 24838 tablet_service.cc:1514] Processing DeleteTablet for tablet c38b898622d34e228f5cbd7c18743fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:24 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:42076
I20260925 11:10:24.831759 24570 tablet_service.cc:1514] Processing DeleteTablet for tablet c38b898622d34e228f5cbd7c18743fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:24 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:37812
I20260925 11:10:24.832113 24704 tablet_service.cc:1514] Processing DeleteTablet for tablet c38b898622d34e228f5cbd7c18743fe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:24 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:41186
I20260925 11:10:24.833268 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24507
I20260925 11:10:24.880029 24946 tablet_replica.cc:330] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: stopping tablet replica
I20260925 11:10:24.883209 24945 tablet_replica.cc:330] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: stopping tablet replica
I20260925 11:10:24.883982 24945 raft_consensus.cc:2229] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:10:24.884750 24945 raft_consensus.cc:2258] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:24.887177 24947 tablet_replica.cc:330] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: stopping tablet replica
I20260925 11:10:24.887192 24945 ts_tablet_manager.cc:1905] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:24.887796 24947 raft_consensus.cc:2229] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:24.888335 24946 raft_consensus.cc:2229] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:24.889165 24947 raft_consensus.cc:2258] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:24.889937 24946 raft_consensus.cc:2258] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:24.892306 24947 ts_tablet_manager.cc:1905] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 11:10:24 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:10:24.894903 24946 ts_tablet_manager.cc:1905] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:24.899173 24947 ts_tablet_manager.cc:1918] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:24.899418 24947 log.cc:1196] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c38b898622d34e228f5cbd7c18743fe8
I20260925 11:10:24.900010 24947 ts_tablet_manager.cc:1939] T c38b898622d34e228f5cbd7c18743fe8 P 259be75582e74ec582a119144f4e42c0: Deleting consensus metadata
I20260925 11:10:24.902652 24431 catalog_manager.cc:4841] TS 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): tablet c38b898622d34e228f5cbd7c18743fe8 (table test-table [id=4b5c0972a3de4134a72e1ead69d555d7]) successfully deleted
I20260925 11:10:24.903149 24945 ts_tablet_manager.cc:1918] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:24.903364 24945 log.cc:1196] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c38b898622d34e228f5cbd7c18743fe8
I20260925 11:10:24.903919 24945 ts_tablet_manager.cc:1939] T c38b898622d34e228f5cbd7c18743fe8 P 96af50b917a74c37b42de23cc8babb0e: Deleting consensus metadata
I20260925 11:10:24.904175 24946 ts_tablet_manager.cc:1918] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:24.904381 24946 log.cc:1196] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c38b898622d34e228f5cbd7c18743fe8
I20260925 11:10:24.905081 24946 ts_tablet_manager.cc:1939] T c38b898622d34e228f5cbd7c18743fe8 P 106bcf16ce3946a999d73b26187ad37a: Deleting consensus metadata
I20260925 11:10:24.905829 24432 catalog_manager.cc:4841] TS 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): tablet c38b898622d34e228f5cbd7c18743fe8 (table test-table [id=4b5c0972a3de4134a72e1ead69d555d7]) successfully deleted
I20260925 11:10:24.907282 24433 catalog_manager.cc:4841] TS 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589): tablet c38b898622d34e228f5cbd7c18743fe8 (table test-table [id=4b5c0972a3de4134a72e1ead69d555d7]) successfully deleted
I20260925 11:10:24.910568 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:24.834790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.8.181.1:33589)
0925 11:10:24.835024 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:24.835033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:24.869165 (+ 34132us) server_negotiation.cc:407] Connection header received
0925 11:10:24.869590 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:24.869600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:24.869777 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:24.869885 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:24.880896 (+ 11011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.882204 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.885092 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:24.885871 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:24.885993 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:24.900993 (+ 15000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:24.901012 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:24.901061 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:24.901124 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:24.907044 (+ 5920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.908044 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.908053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.908056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.908210 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:24.908661 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:24.908666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:24.908668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:24.909144 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:24.909490 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:24.909864 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 11:10:24.910113 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:10:24.911278 24630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:25.052307 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24507
I20260925 11:10:25.070580 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24641
Sep 25 11:10:25 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:10:25.089660 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:25.071450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.8.181.2:46443)
0925 11:10:25.071611 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:25.071616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:25.071633 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:10:25.071832 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:25.071836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:25.071921 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:25.072005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:25.073762 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:25.074793 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:25.076810 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:25.077183 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:25.077280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:25.083004 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:25.083022 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:25.083038 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:25.083081 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:25.086194 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:25.087229 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:25.087237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:25.087239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:25.087376 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:25.087834 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:25.087840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:25.087866 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:25.088403 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:25.088726 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:25.089045 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 11:10:25.089285 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:10:25.090405 24764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:25.184019 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24641
I20260925 11:10:25.207641 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24775
Sep 25 11:10:25 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:10:25.227427 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:25.208602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.8.181.3:44023)
0925 11:10:25.208799 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:25.208804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:25.208943 (+ 139us) server_negotiation.cc:407] Connection header received
0925 11:10:25.209127 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:25.209132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:25.209204 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:25.209329 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:25.211213 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:25.212014 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:25.214042 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:25.214598 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:25.214710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:25.220686 (+ 5976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:25.220703 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:25.220720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:25.220781 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:25.224130 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:25.225041 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:25.225052 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:25.225055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:25.225213 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:25.225635 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:25.225643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:25.225650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:25.226019 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:25.226387 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:25.226783 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 11:10:25.227028 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:10:25.228338 24898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:25.359798 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24775
I20260925 11:10:25.380551 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24414
I20260925 11:10:25.381819 24476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:25.495326 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24414
I20260925 11:10:45.517344 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--webserver_interface=127.8.181.62
--webserver_port=32847
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:10:45.859572 24953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:45.859972 24953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:45.860157 24953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:45.867918 24953 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 11:10:45.868078 24953 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 11:10:45.868176 24953 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 11:10:45.868273 24953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:10:45.868376 24953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:45.868498 24953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:45.868588 24953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:10:45.868669 24953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:10:45.879024 24953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=32847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:45.880404 24953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:45.881795 24953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:45.904767 24959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:45.904904 24960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:45.904908 24962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:45.905114 24953 server_base.cc:1034] running on GCE node
I20260925 11:10:45.906368 24953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:45.908051 24953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:45.909310 24953 hybrid_clock.cc:648] HybridClock initialized: now 1790334645909227 us; error 63 us; skew 500 ppm
I20260925 11:10:45.909721 24953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:45 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334645, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:45.913509 24953 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:10:45.915249 24953 webserver.cc:460] Webserver started at http://127.8.181.62:32847/ using document root <none> and password file <none>
I20260925 11:10:45.916029 24953 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:45.916136 24953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:45.922349 24953 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 11:10:45.925598 24969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:45.927186 24953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:10:45.927336 24953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d6a89672f58b40b0b85dde258173a8ee"
format_stamp: "Formatted at 2026-09-25 11:10:22 on dist-test-slave-6042"
server_key: "9d9ed6fa461ae22221a1bd56d1de9a9d"
server_key_iv: "3aeb7bda4639555c39be43f6c516aa70"
server_key_version: "encryptionkey@0"
I20260925 11:10:45.928359 24953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:45.945582 24953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:45.950030 24953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:45.950285 24953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:45.970109 24953 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:36663
I20260925 11:10:45.970142 25020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:36663 every 8 connection(s)
I20260925 11:10:45.971745 24953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:10:45.974942 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 24953
I20260925 11:10:45.975971 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:33589
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=45739
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:10:45.981528 25021 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:10:45.986757 25021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap starting.
I20260925 11:10:46.008141 25021 log.cc:824] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:46.020965 25021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:10:46.021600 25021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap complete.
I20260925 11:10:46.031220 25021 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:46.032229 25021 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Initialized, Role: FOLLOWER
I20260925 11:10:46.032938 25021 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:46.033195 25021 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:10:46.033327 25021 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:10:46.033516 25021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 1 FOLLOWER]: Advancing to term 2
I20260925 11:10:46.037720 25021 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:46.038280 25021 leader_election.cc:302] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a89672f58b40b0b85dde258173a8ee; no voters:
I20260925 11:10:46.038967 25021 leader_election.cc:288] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 11:10:46.039126 25025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Leader election won for term 2
I20260925 11:10:46.040390 25025 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 LEADER]: Becoming Leader. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Running, Role: LEADER
I20260925 11:10:46.041029 25025 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:10:46.041416 25021 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:10:46.043829 25027 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a89672f58b40b0b85dde258173a8ee. Latest consensus state: current_term: 2 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:10:46.043802 25026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:10:46.044269 25026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:46.044266 25027 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:10:46.046622 25031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:10:46.054965 25031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:10:46.056559 25031 catalog_manager.cc:1259] Loaded cluster ID: 48feb9b1252a42f08b052d9afc799109
I20260925 11:10:46.056721 25031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:10:46.060977 25031 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:10:46.062685 25031 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Loaded TSK: 0
I20260925 11:10:46.093457 25031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Generated new TSK 1
I20260925 11:10:46.093978 25031 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 11:10:46.346398 25023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:46.346859 25023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:46.347047 25023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:46.355470 25023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:46.355644 25023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:46.355823 25023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:10:46.366704 25023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=45739
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:46.368130 25023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:46.369486 25023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:46.384119 25051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:46.384059 25049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:46.384217 25048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:46.384848 25023 server_base.cc:1034] running on GCE node
I20260925 11:10:46.385519 25023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:46.390698 25023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:46.392009 25023 hybrid_clock.cc:648] HybridClock initialized: now 1790334646391944 us; error 90 us; skew 500 ppm
I20260925 11:10:46.392412 25023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:46 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334646, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:46.396865 25023 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:10:46.398890 25023 webserver.cc:460] Webserver started at http://127.8.181.1:45739/ using document root <none> and password file <none>
I20260925 11:10:46.399822 25023 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:46.399986 25023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:46.407074 25023 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 11:10:46.410579 25058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.412376 25023 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:10:46.412556 25023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "106bcf16ce3946a999d73b26187ad37a"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "ed13808c88eb98c3948ebb46650d06a2"
server_key_iv: "75c639a37dbe98b66c326ab8d3efaf92"
server_key_version: "encryptionkey@0"
I20260925 11:10:46.413900 25023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:46.445205 25023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:46.450947 25023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:46.451247 25023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:46.452462 25023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:46.454171 25023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:46.454278 25023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.454367 25023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:46.454438 25023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.489907 25023 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33589
I20260925 11:10:46.490007 25170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33589 every 8 connection(s)
I20260925 11:10:46.491437 25023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:10:46.495235 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25023
I20260925 11:10:46.496001 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:46443
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=41733
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:10:46 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334646, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:46.518322 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:46.494253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:51663 (local address 127.8.181.62:36663)
0925 11:10:46.494944 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:46.494965 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:46.495648 (+ 683us) server_negotiation.cc:407] Connection header received
0925 11:10:46.498826 (+ 3178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:46.498838 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:46.499181 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:46.499474 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:46.501708 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.503057 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.505248 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.506390 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.506621 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:46.510509 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:46.510525 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:46.510549 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:46.510595 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:46.513551 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:46.514579 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:46.514586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:46.514591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:46.514746 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:46.515247 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:46.515250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:46.515252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:46.515780 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:46.515994 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:46.516982 (+ 988us) server_negotiation.cc:299] Negotiation successful
0925 11:10:46.517313 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
I20260925 11:10:46.518977 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:46.494523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:51663)
0925 11:10:46.495448 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:46.495497 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:46.498227 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:46.499681 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:46.499694 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:46.500535 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:46.501472 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:46.501502 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.503233 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.503242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:46.505060 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:46.505073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.506685 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.506691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:46.506891 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:46.507586 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:46.507613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:46.510156 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:46.513711 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:46.513723 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:46.513741 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:46.514419 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:46.514877 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:46.514885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:46.514892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:46.515141 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:46.515987 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:46.515999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:46.516474 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 11:10:46.516824 (+ 350us) client_negotiation.cc:240] Negotiation successful
0925 11:10:46.517476 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":378,"threads_started":1}
I20260925 11:10:46.522018 25171 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:46.522327 25171 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:46.523064 25171 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:46.526137 24986 ts_manager.cc:194] Registered new tserver with Master: 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589)
I20260925 11:10:46.528448 24986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:51663
W20260925 11:10:46.840312 25175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:46.840608 25175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:46.840768 25175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:46.848238 25175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:46.848362 25175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:46.848548 25175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:10:46.858496 25175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:46443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=41733
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:46.859820 25175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:46.861135 25175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:46.874354 25181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:46.874292 25180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:46.874485 25183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:46.875020 25175 server_base.cc:1034] running on GCE node
I20260925 11:10:46.875636 25175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:46.876601 25175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:46.877868 25175 hybrid_clock.cc:648] HybridClock initialized: now 1790334646877786 us; error 68 us; skew 500 ppm
I20260925 11:10:46.878227 25175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:46 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334646, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:46.881414 25175 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:10:46.883036 25175 webserver.cc:460] Webserver started at http://127.8.181.2:41733/ using document root <none> and password file <none>
I20260925 11:10:46.883728 25175 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:46.883891 25175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:46.889403 25175 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:10:46.892350 25190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.893783 25175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:10:46.893970 25175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "259be75582e74ec582a119144f4e42c0"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "eca5b99d37191d40f2cb939f2f9832d7"
server_key_iv: "6b32821f1ffd18653e9a44d0e93d0d24"
server_key_version: "encryptionkey@0"
I20260925 11:10:46.895056 25175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:46.907495 25175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:46.911953 25175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:46.912240 25175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:46.913249 25175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:46.914707 25175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:46.914831 25175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.914963 25175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:46.915051 25175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:46.950223 25175 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:46443
I20260925 11:10:46.950313 25302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:46443 every 8 connection(s)
I20260925 11:10:46.951838 25175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:10:46.953662 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25175
I20260925 11:10:46.954504 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:44023
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=33431
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 11:10:46 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334646, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:46.975513 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:46.954438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35867 (local address 127.8.181.62:36663)
0925 11:10:46.954664 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:46.954671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:46.956121 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 11:10:46.958007 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:46.958013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:46.958109 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:46.958220 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:46.959856 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.960554 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.962878 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.963377 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.963498 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:46.968043 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:46.968049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:46.968052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:46.968088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:46.971491 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:46.972564 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:46.972570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:46.972574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:46.972715 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:46.973307 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:46.973315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:46.973321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:46.973714 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:46.973872 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:46.974883 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0925 11:10:46.975144 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 11:10:46.977049 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:46.955180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:35867)
0925 11:10:46.955949 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:46.955991 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:46.957716 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:46.958515 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:46.958530 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:46.959087 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:46.959662 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:46.959678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.960712 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.960724 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:46.962692 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:46.962709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:46.963621 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:46.963630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:46.963974 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:46.964743 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:46.964777 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:46.967690 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:46.971679 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:46.971693 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:46.971713 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:46.972407 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:46.972856 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:46.972863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:46.972870 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:46.973159 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:46.973879 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:46.973891 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:46.974414 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 11:10:46.974780 (+ 366us) client_negotiation.cc:240] Negotiation successful
0925 11:10:46.975424 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":314,"threads_started":1}
I20260925 11:10:46.978649 25303 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:46.979015 25303 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:46.979959 25303 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:46.981745 24986 ts_manager.cc:194] Registered new tserver with Master: 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443)
I20260925 11:10:46.982869 24986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:35867
W20260925 11:10:47.294221 25306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:10:47.294605 25306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:10:47.294780 25306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:10:47.302544 25306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:10:47.302702 25306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:10:47.302874 25306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:10:47.313113 25306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=33431
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:10:47.314529 25306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:10:47.315848 25306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:10:47.329751 25311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:47.329798 25312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:10:47.329821 25314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:10:47.330793 25306 server_base.cc:1034] running on GCE node
I20260925 11:10:47.331502 25306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:10:47.332599 25306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:10:47.333827 25306 hybrid_clock.cc:648] HybridClock initialized: now 1790334647333768 us; error 51 us; skew 500 ppm
I20260925 11:10:47.334154 25306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:10:47 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:10:47.337527 25306 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:10:47.339123 25306 webserver.cc:460] Webserver started at http://127.8.181.3:33431/ using document root <none> and password file <none>
I20260925 11:10:47.339818 25306 fs_manager.cc:359] Metadata directory not provided
I20260925 11:10:47.339984 25306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:10:47.345676 25306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:10:47.348490 25321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:47.349900 25306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:10:47.350082 25306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "96af50b917a74c37b42de23cc8babb0e"
format_stamp: "Formatted at 2026-09-25 11:10:24 on dist-test-slave-6042"
server_key: "0fecb597ea5bda4869d80fee27945bd3"
server_key_iv: "529c5fa0b6d0f68127ea999e379edc86"
server_key_version: "encryptionkey@0"
I20260925 11:10:47.351253 25306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:10:47.386991 25306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:10:47.391847 25306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:10:47.392141 25306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:10:47.393153 25306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:10:47.394579 25306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:10:47.394699 25306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:47.394836 25306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:10:47.394923 25306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:10:47.430327 25306 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:44023
I20260925 11:10:47.430377 25433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:44023 every 8 connection(s)
I20260925 11:10:47.431993 25306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:10:47.432539 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25306
Sep 25 11:10:47 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334647, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:10:47.451237 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.433705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.8.181.62:36663)
0925 11:10:47.433899 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.433905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.434591 (+ 686us) server_negotiation.cc:407] Connection header received
0925 11:10:47.434835 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.434841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.434969 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.435110 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:47.437565 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.438739 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.440287 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.440930 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.441070 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.444495 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:47.444505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:47.444511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:47.444562 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:47.447874 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.448787 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.448795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.448799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.448982 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.449452 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.449463 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.449465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.449812 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:47.450159 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.450460 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.450733 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":2816}
I20260925 11:10:47.457388 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.435159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:53269 (local address 127.8.181.62:36663)
0925 11:10:47.435647 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.435651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.436821 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 11:10:47.439255 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.439260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.439349 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.439467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:47.441370 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.442852 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.444617 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.445238 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.445363 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.449432 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:47.449437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:47.449441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:47.449473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:47.453620 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.454786 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.454794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.454798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.454936 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.455548 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.455552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.455554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.455903 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:47.456027 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.456847 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.457066 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260925 11:10:47.458426 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.435778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:53269)
0925 11:10:47.436648 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.436693 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.439003 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.439723 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.439732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.440322 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:10:47.441156 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.441182 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.443064 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.443069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.444430 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.444445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.445483 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.445492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.445762 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.446418 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:10:47.446453 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:10:47.449169 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:10:47.453789 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:47.453800 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:47.453821 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:47.454641 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:47.455085 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:10:47.455091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:10:47.455095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:10:47.455348 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:10:47.456074 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:10:47.456081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:10:47.456454 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.456706 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.457200 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":371,"threads_started":1}
I20260925 11:10:47.460075 25434 heartbeater.cc:347] Connected to a master server at 127.8.181.62:36663
I20260925 11:10:47.460431 25434 heartbeater.cc:464] Registering TS with master...
I20260925 11:10:47.461187 25434 heartbeater.cc:511] Master 127.8.181.62:36663 requested a full tablet report, sending...
I20260925 11:10:47.462882 24986 ts_manager.cc:194] Registered new tserver with Master: 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
I20260925 11:10:47.463721 24986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:53269
I20260925 11:10:47.467219 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:10:47.474370 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.468135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.8.181.62:36663)
0925 11:10:47.468314 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.468318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.468835 (+ 517us) server_negotiation.cc:407] Connection header received
0925 11:10:47.469021 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.469025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.469167 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.469292 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:47.470007 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.470896 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.472525 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.472998 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.473095 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.473634 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:47.473864 (+ 230us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 11:10:47.474170 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50954: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 11:10:47.474547 25437 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50954: token expired
W20260925 11:10:47.475358 24909 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.181.62:36663)
I20260925 11:10:47.488821 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.476000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.8.181.62:36663)
0925 11:10:47.476207 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.476214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.476232 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:10:47.476310 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.476313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.476369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.476486 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:47.478118 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.478958 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.480346 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.480751 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.480861 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.483675 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:47.483681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:47.483684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:47.483719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:47.486325 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.487084 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.487088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.487091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.487212 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.487537 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.487540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.487542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.487859 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:47.488061 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.488311 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.488484 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:10:47.492790 24986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:10:47.500626 24986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:10:47.529162 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.514145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.8.181.1:33589)
0925 11:10:47.518673 (+ 4528us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.518685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.518725 (+ 40us) server_negotiation.cc:407] Connection header received
0925 11:10:47.518888 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.518900 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.519374 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.520622 (+ 1248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:47.520884 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.522602 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.525928 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.527707 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.527937 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.528149 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.528587 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.528785 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4270,"thread_start_us":4097,"threads_started":1}
I20260925 11:10:47.529196 25444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.513779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55678)
0925 11:10:47.515109 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.515164 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.515368 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.519700 (+ 4332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.519712 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.519763 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:47.520323 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.520360 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.522777 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.522783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.525685 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.525702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.528037 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.528043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.528283 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.528346 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.528500 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.528722 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":63,"thread_start_us":1826,"threads_started":1}
I20260925 11:10:47.530012 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.515013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.8.181.3:44023)
0925 11:10:47.516862 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.516876 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.519871 (+ 2995us) server_negotiation.cc:407] Connection header received
0925 11:10:47.520140 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.520146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.520367 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.520566 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:47.521353 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.523095 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.525911 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.527707 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.527871 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.528145 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.529406 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.529591 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":1429,"threads_started":1}
I20260925 11:10:47.529256 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.515469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44023 (local address 127.0.0.1:42032)
0925 11:10:47.519759 (+ 4290us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.519777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.519964 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.520693 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.520699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.520727 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:47.521153 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.521165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.523246 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.523253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.525685 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.525702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.528315 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.528327 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.528459 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.528533 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.528633 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.528783 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4172,"mutex_wait_us":23,"thread_start_us":4027,"threads_started":1}
I20260925 11:10:47.530511 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.514115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46443 (local address 127.0.0.1:47332)
0925 11:10:47.519356 (+ 5241us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.519377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.519702 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.520380 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.520386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.520420 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:47.520869 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.520880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.523219 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.523229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.527679 (+ 4450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.527692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.529700 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.529706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.529885 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.529912 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.530014 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.530148 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4872,"thread_start_us":4732,"threads_started":1}
I20260925 11:10:47.531636 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.514839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.8.181.2:46443)
0925 11:10:47.518004 (+ 3165us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.518017 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.519634 (+ 1617us) server_negotiation.cc:407] Connection header received
0925 11:10:47.519920 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.519929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.520166 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.520395 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:47.521038 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.523038 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.527857 (+ 4819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.529440 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.529578 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.529786 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.530068 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.530232 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2928,"thread_start_us":2763,"threads_started":1}
I20260925 11:10:47.532557 25171 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:47.534317 25369 tablet_service.cc:1467] Processing CreateTablet for tablet 6701b05d1914438db2c91ceee76e87f3 (DEFAULT_TABLE table=test-table [id=547e60cce3b14d3294e6f3b8adb02091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:47.534322 25106 tablet_service.cc:1467] Processing CreateTablet for tablet 6701b05d1914438db2c91ceee76e87f3 (DEFAULT_TABLE table=test-table [id=547e60cce3b14d3294e6f3b8adb02091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:47.535920 25369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6701b05d1914438db2c91ceee76e87f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:47.535418 25238 tablet_service.cc:1467] Processing CreateTablet for tablet 6701b05d1914438db2c91ceee76e87f3 (DEFAULT_TABLE table=test-table [id=547e60cce3b14d3294e6f3b8adb02091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:10:47.536196 25106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6701b05d1914438db2c91ceee76e87f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:47.536820 25238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6701b05d1914438db2c91ceee76e87f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:10:47.549430 25450 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Bootstrap starting.
I20260925 11:10:47.553629 25451 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Bootstrap starting.
I20260925 11:10:47.553949 25450 tablet_bootstrap.cc:654] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:47.555393 25450 log.cc:824] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:47.555637 25452 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Bootstrap starting.
I20260925 11:10:47.557622 25451 tablet_bootstrap.cc:654] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:47.558732 25451 log.cc:824] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:47.559741 25452 tablet_bootstrap.cc:654] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Neither blocks nor log segments found. Creating new log.
I20260925 11:10:47.559877 25450 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: No bootstrap required, opened a new log
I20260925 11:10:47.560242 25450 ts_tablet_manager.cc:1397] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260925 11:10:47.561113 25452 log.cc:824] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Log is configured to *not* fsync() on all Append() calls
I20260925 11:10:47.561692 25451 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: No bootstrap required, opened a new log
I20260925 11:10:47.562058 25451 ts_tablet_manager.cc:1397] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 11:10:47.564355 25452 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: No bootstrap required, opened a new log
I20260925 11:10:47.564639 25452 ts_tablet_manager.cc:1397] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 11:10:47.569242 25451 raft_consensus.cc:348] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.569645 25451 raft_consensus.cc:374] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:47.569782 25451 raft_consensus.cc:729] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106bcf16ce3946a999d73b26187ad37a, State: Initialized, Role: FOLLOWER
I20260925 11:10:47.570422 25451 consensus_queue.cc:260] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.572582 25450 raft_consensus.cc:348] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.573130 25450 raft_consensus.cc:374] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:47.573268 25450 raft_consensus.cc:729] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96af50b917a74c37b42de23cc8babb0e, State: Initialized, Role: FOLLOWER
I20260925 11:10:47.574155 25451 ts_tablet_manager.cc:1428] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260925 11:10:47.574062 25450 consensus_queue.cc:260] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.576160 25450 ts_tablet_manager.cc:1428] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 11:10:47.576718 25434 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:47.576489 25452 raft_consensus.cc:348] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.577023 25452 raft_consensus.cc:374] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:10:47.577217 25452 raft_consensus.cc:729] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259be75582e74ec582a119144f4e42c0, State: Initialized, Role: FOLLOWER
I20260925 11:10:47.578001 25452 consensus_queue.cc:260] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.579206 25303 heartbeater.cc:503] Master 127.8.181.62:36663 was elected leader, sending a full tablet report...
I20260925 11:10:47.579999 25452 ts_tablet_manager.cc:1428] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260925 11:10:47.636556 25458 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:10:47.636901 25458 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.638687 25458 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
I20260925 11:10:47.646909 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.639313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:37509 (local address 127.8.181.3:44023)
0925 11:10:47.639574 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.639583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.640341 (+ 758us) server_negotiation.cc:407] Connection header received
0925 11:10:47.640505 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.640510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.640592 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.640684 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:47.641340 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.642196 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.644788 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.645845 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.646020 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.646131 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.646514 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.646643 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:10:47.647086 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.638955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44023 (local address 127.8.181.2:37509)
0925 11:10:47.640214 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.640234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.640360 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.640714 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.640719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.640743 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:47.641180 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.641190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.642324 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.642329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.644613 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.644629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.646079 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.646085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.646284 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.646328 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.646488 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.646629 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":43,"thread_start_us":977,"threads_started":1}
I20260925 11:10:47.647109 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.639175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42035 (local address 127.8.181.1:33589)
0925 11:10:47.639456 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.639465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.640488 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 11:10:47.640704 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.640709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.640784 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.640883 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:10:47.641480 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.642515 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.644953 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.646024 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.646225 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.646334 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.646720 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.646849 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:10:47.647240 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.638999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.8.181.2:42035)
0925 11:10:47.640340 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.640359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:10:47.640547 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:10:47.640917 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:10:47.640921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:10:47.640950 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:10:47.641322 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.641333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.642674 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.642677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.644808 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:10:47.644821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.646287 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.646293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:10:47.646559 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.646584 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 11:10:47.646694 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 11:10:47.646826 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1203,"mutex_wait_us":45,"thread_start_us":783,"threads_started":1}
I20260925 11:10:47.648361 25389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6701b05d1914438db2c91ceee76e87f3" candidate_uuid: "259be75582e74ec582a119144f4e42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96af50b917a74c37b42de23cc8babb0e" is_pre_election: true
I20260925 11:10:47.648487 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6701b05d1914438db2c91ceee76e87f3" candidate_uuid: "259be75582e74ec582a119144f4e42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106bcf16ce3946a999d73b26187ad37a" is_pre_election: true
I20260925 11:10:47.648818 25389 raft_consensus.cc:2454] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259be75582e74ec582a119144f4e42c0 in term 0.
I20260925 11:10:47.649109 25126 raft_consensus.cc:2454] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259be75582e74ec582a119144f4e42c0 in term 0.
I20260925 11:10:47.649483 25193 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e; no voters:
I20260925 11:10:47.649964 25458 raft_consensus.cc:2781] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:10:47.650112 25458 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:10:47.650213 25458 raft_consensus.cc:3037] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:47.652474 25458 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.653057 25458 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [CANDIDATE]: Term 1 election: Requested vote from peers 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
I20260925 11:10:47.653548 25126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6701b05d1914438db2c91ceee76e87f3" candidate_uuid: "259be75582e74ec582a119144f4e42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "106bcf16ce3946a999d73b26187ad37a"
I20260925 11:10:47.653545 25389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6701b05d1914438db2c91ceee76e87f3" candidate_uuid: "259be75582e74ec582a119144f4e42c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96af50b917a74c37b42de23cc8babb0e"
I20260925 11:10:47.653766 25389 raft_consensus.cc:3037] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:47.653762 25126 raft_consensus.cc:3037] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:10:47.657059 25126 raft_consensus.cc:2454] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259be75582e74ec582a119144f4e42c0 in term 1.
I20260925 11:10:47.657055 25389 raft_consensus.cc:2454] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259be75582e74ec582a119144f4e42c0 in term 1.
I20260925 11:10:47.657491 25193 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a, 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e; no voters:
I20260925 11:10:47.657811 25458 raft_consensus.cc:2781] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:10:47.658249 25458 raft_consensus.cc:686] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 1 LEADER]: Becoming Leader. State: Replica: 259be75582e74ec582a119144f4e42c0, State: Running, Role: LEADER
I20260925 11:10:47.658758 25458 consensus_queue.cc:237] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:10:47.663166 24986 catalog_manager.cc:5495] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259be75582e74ec582a119144f4e42c0 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "259be75582e74ec582a119144f4e42c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } health_report { overall_health: UNKNOWN } } }
W20260925 11:10:47.684233 25435 tablet.cc:2369] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:10:47.688038 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.682656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.8.181.2:46443)
0925 11:10:47.682905 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.682910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.682928 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:10:47.683127 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.683131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.683213 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.683333 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:10:47.683661 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.684432 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.685949 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.686323 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.686419 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.686937 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:10:47.687438 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:10:47.687543 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.687665 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.687732 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 11:10:47.696151 25389 raft_consensus.cc:1261] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Refusing update from remote peer 259be75582e74ec582a119144f4e42c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:47.696179 25126 raft_consensus.cc:1261] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Refusing update from remote peer 259be75582e74ec582a119144f4e42c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:10:47.697190 25461 consensus_queue.cc:1035] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:10:47.697566 25458 consensus_queue.cc:1035] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 11:10:47.703775 25304 tablet.cc:2369] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:10:47.708935 25464 mvcc.cc:205] Tried to move back new op lower bound from 7333210716953247744 to 7333210716814147584. Current Snapshot: MvccSnapshot[applied={T|T < 7333210716953247744}]
I20260925 11:10:47.710428 25465 mvcc.cc:205] Tried to move back new op lower bound from 7333210716953247744 to 7333210716814147584. Current Snapshot: MvccSnapshot[applied={T|T < 7333210716953247744}]
I20260925 11:10:47.729050 24985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:47.729398 24985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:10:47.733425 24985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Sending DeleteTablet for 3 replicas of tablet 6701b05d1914438db2c91ceee76e87f3
I20260925 11:10:47.734632 25238 tablet_service.cc:1514] Processing DeleteTablet for tablet 6701b05d1914438db2c91ceee76e87f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:47332
I20260925 11:10:47.734643 25106 tablet_service.cc:1514] Processing DeleteTablet for tablet 6701b05d1914438db2c91ceee76e87f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:55678
I20260925 11:10:47.734719 25369 tablet_service.cc:1514] Processing DeleteTablet for tablet 6701b05d1914438db2c91ceee76e87f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:10:47 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:42032
I20260925 11:10:47.737763 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25023
I20260925 11:10:47.747334 25474 tablet_replica.cc:330] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: stopping tablet replica
I20260925 11:10:47.748049 25474 raft_consensus.cc:2229] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:10:47.748740 25474 raft_consensus.cc:2258] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:47.751257 25474 ts_tablet_manager.cc:1905] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:47.757542 25476 tablet_replica.cc:330] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: stopping tablet replica
I20260925 11:10:47.758608 25476 raft_consensus.cc:2229] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:10:47.759466 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.738739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.8.181.1:33589)
0925 11:10:47.738966 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.738975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.739002 (+ 27us) server_negotiation.cc:407] Connection header received
0925 11:10:47.739156 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.739165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.739299 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.739449 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:47.741715 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.743396 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.745995 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.746975 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.747140 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.750683 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:47.750703 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:47.750729 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:47.750806 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:47.756043 (+ 5237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.757056 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.757071 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.757077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.757257 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.757725 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.757731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.757735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.758144 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:47.758512 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.758829 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.759065 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:10:47.760146 25166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:47.760131 25476 raft_consensus.cc:2258] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:10:47.760488 25474 ts_tablet_manager.cc:1918] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:47.760809 25474 log.cc:1196] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/6701b05d1914438db2c91ceee76e87f3
I20260925 11:10:47.761822 25474 ts_tablet_manager.cc:1939] T 6701b05d1914438db2c91ceee76e87f3 P 96af50b917a74c37b42de23cc8babb0e: Deleting consensus metadata
I20260925 11:10:47.764441 24972 catalog_manager.cc:4841] TS 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): tablet 6701b05d1914438db2c91ceee76e87f3 (table test-table [id=547e60cce3b14d3294e6f3b8adb02091]) successfully deleted
I20260925 11:10:47.764683 25476 ts_tablet_manager.cc:1905] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:10:47.771158 25476 ts_tablet_manager.cc:1918] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:10:47.771358 25476 log.cc:1196] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/6701b05d1914438db2c91ceee76e87f3
I20260925 11:10:47.771932 25476 ts_tablet_manager.cc:1939] T 6701b05d1914438db2c91ceee76e87f3 P 259be75582e74ec582a119144f4e42c0: Deleting consensus metadata
I20260925 11:10:47.773658 24971 catalog_manager.cc:4841] TS 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): tablet 6701b05d1914438db2c91ceee76e87f3 (table test-table [id=547e60cce3b14d3294e6f3b8adb02091]) successfully deleted
I20260925 11:10:47.925421 25475 tablet_replica.cc:330] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: stopping tablet replica
I20260925 11:10:47.925510 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25023
I20260925 11:10:47.943881 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25175
W20260925 11:10:47.943881 24973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.181.1:33589 (error 108)
I20260925 11:10:47.945979 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.945426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55702)
0925 11:10:47.945641 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.945782 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 11:10:47.946303 24973 catalog_manager.cc:4584] TS 106bcf16ce3946a999d73b26187ad37a (127.8.181.1:33589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6701b05d1914438db2c91ceee76e87f3: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
I20260925 11:10:47.960062 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.945216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.8.181.2:46443)
0925 11:10:47.945428 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:47.945434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:47.945455 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:10:47.945525 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:47.945534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:47.945619 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:47.945736 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:47.947332 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.948226 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.949661 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:47.950213 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:47.950359 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:47.953394 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:47.953406 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:47.953419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:47.953457 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:47.957017 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.957762 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.957772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.957781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.957940 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:47.958336 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:47.958341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:47.958344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:47.958777 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:47.959110 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:47.959428 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 11:10:47.959654 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 11:10:47.960871 25298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:48.000008 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:47.999551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55704)
0925 11:10:47.999761 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:47.999866 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 11:10:48.001140 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.000770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55716)
0925 11:10:48.000919 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.001027 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 11:10:48.061956 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.061474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55724)
0925 11:10:48.061707 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.061793 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 11:10:48.063081 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.062767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55736)
0925 11:10:48.062896 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.062970 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 11:10:48.090829 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25175
I20260925 11:10:48.110749 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25306
I20260925 11:10:48.124822 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.111637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.8.181.3:44023)
0925 11:10:48.111817 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 11:10:48.111822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:10:48.111840 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:10:48.112057 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:10:48.112062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:10:48.112128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:10:48.112217 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:10:48.113648 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:48.114400 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:48.115678 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:10:48.116101 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:10:48.116195 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:10:48.119027 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:10:48.119042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:10:48.119058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:10:48.119097 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:10:48.121885 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:48.122668 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:48.122676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:48.122678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:48.122818 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:10:48.123213 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:10:48.123216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:10:48.123218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:10:48.123547 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:10:48.123764 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 11:10:48.124237 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 11:10:48.124487 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 11:10:48.125554 25429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:48.143990 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.143474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55748)
0925 11:10:48.143688 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.143777 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 11:10:48.145109 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.144709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55756)
0925 11:10:48.144913 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.144991 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 11:10:48.218690 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25306
I20260925 11:10:48.235736 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 24953
I20260925 11:10:48.236900 25016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:10:48.352279 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 24953
I20260925 11:10:48.352460 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:10:48.351916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:33589 (local address 127.0.0.1:55764)
0925 11:10:48.352130 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 11:10:48.352233 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 11:11:08.374686 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--webserver_interface=127.8.181.62
--webserver_port=32847
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:08.740788 25482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:08.741168 25482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:08.741340 25482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:08.749691 25482 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 11:11:08.749843 25482 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 11:11:08.749925 25482 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 11:11:08.750006 25482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:08.750090 25482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:08.750193 25482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:08.750284 25482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:08.750365 25482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:08.762228 25482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=32847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:08.763756 25482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:08.765228 25482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:08.778945 25488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:08.779083 25487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:08.779126 25490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:08.779367 25482 server_base.cc:1034] running on GCE node
I20260925 11:11:08.780579 25482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:08.782249 25482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:08.783514 25482 hybrid_clock.cc:648] HybridClock initialized: now 1790334668783453 us; error 66 us; skew 500 ppm
I20260925 11:11:08.783900 25482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:08 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:08.787448 25482 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:11:08.789208 25482 webserver.cc:460] Webserver started at http://127.8.181.62:32847/ using document root <none> and password file <none>
I20260925 11:11:08.790104 25482 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:08.790251 25482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:08.796541 25482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:11:08.799954 25497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:08.801640 25482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 11:11:08.801883 25482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d6a89672f58b40b0b85dde258173a8ee"
format_stamp: "Formatted at 2026-09-25 11:10:22 on dist-test-slave-6042"
server_key: "9d9ed6fa461ae22221a1bd56d1de9a9d"
server_key_iv: "3aeb7bda4639555c39be43f6c516aa70"
server_key_version: "encryptionkey@0"
I20260925 11:11:08.803117 25482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:08.832232 25482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:08.837332 25482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:08.837661 25482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:08.858141 25482 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:36663
I20260925 11:11:08.858192 25548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:36663 every 8 connection(s)
I20260925 11:11:08.859733 25482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:08.861660 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25482
I20260925 11:11:08.862589 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:33589
--local_ip_for_outbound_sockets=127.8.181.1
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=45739
--webserver_interface=127.8.181.1
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:11:08.868376 25549 sys_catalog.cc:263] Verifying existing consensus state
I20260925 11:11:08.873708 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap starting.
I20260925 11:11:08.900701 25549 log.cc:824] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:08.917927 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:11:08.918617 25549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Bootstrap complete.
I20260925 11:11:08.928304 25549 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:11:08.929330 25549 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Initialized, Role: FOLLOWER
I20260925 11:11:08.929996 25549 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:11:08.930248 25549 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:08.930385 25549 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:08.930603 25549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 2 FOLLOWER]: Advancing to term 3
I20260925 11:11:08.934768 25549 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:11:08.935335 25549 leader_election.cc:302] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a89672f58b40b0b85dde258173a8ee; no voters:
I20260925 11:11:08.936056 25549 leader_election.cc:288] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 11:11:08.936204 25553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 3 FOLLOWER]: Leader election won for term 3
I20260925 11:11:08.937671 25553 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [term 3 LEADER]: Becoming Leader. State: Replica: d6a89672f58b40b0b85dde258173a8ee, State: Running, Role: LEADER
I20260925 11:11:08.938344 25553 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } }
I20260925 11:11:08.938609 25549 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:08.941154 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a89672f58b40b0b85dde258173a8ee. Latest consensus state: current_term: 3 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:11:08.941615 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:08.941591 25554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6a89672f58b40b0b85dde258173a8ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a89672f58b40b0b85dde258173a8ee" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36663 } } }
I20260925 11:11:08.942014 25554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:08.954176 25562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:08.963351 25562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:08.965009 25562 catalog_manager.cc:1259] Loaded cluster ID: 48feb9b1252a42f08b052d9afc799109
I20260925 11:11:08.965207 25562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:08.969868 25562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:08.971810 25562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Loaded TSK: 0
I20260925 11:11:08.972047 25562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6a89672f58b40b0b85dde258173a8ee: Loaded TSK: 1
F20260925 11:11:08.973256 25562 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7437f699ad google::LogMessage::Fail() at ??:0
@ 0x7f7437f6c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7437f694de google::LogMessage::Flush() at ??:0
@ 0x7f7437f6aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f743bccd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f743bcbc73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f744d99c1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f744d99b783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f744da272a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f744da270c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f744da715e9 std::function<>::operator()() at ??:0
@ 0x7f744d99fcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f744d99dc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f744da25610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f744da25439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f743abb5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f743abb891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f743abb8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f743ab93eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f743de6d6db start_thread at ??:0
@ 0x7f7436e0361f clone at ??:0
W20260925 11:11:09.291919 25551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:09.292236 25551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:09.292379 25551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:09.300065 25551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:09.300194 25551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:09.300381 25551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:09.310932 25551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=45739
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:09.312361 25551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:09.313699 25551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:09.326953 25599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:09.326975 25597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:09.327072 25596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:09.327941 25551 server_base.cc:1034] running on GCE node
I20260925 11:11:09.328612 25551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:09.329838 25551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:09.331070 25551 hybrid_clock.cc:648] HybridClock initialized: now 1790334669330999 us; error 63 us; skew 500 ppm
I20260925 11:11:09.331449 25551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:09 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334669, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:09.334890 25551 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:11:09.336562 25551 webserver.cc:460] Webserver started at http://127.8.181.1:45739/ using document root <none> and password file <none>
I20260925 11:11:09.337322 25551 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:09.337483 25551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:09.343278 25551 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:11:09.346344 25606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:09.347980 25551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:11:09.348186 25551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "106bcf16ce3946a999d73b26187ad37a"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "ed13808c88eb98c3948ebb46650d06a2"
server_key_iv: "75c639a37dbe98b66c326ab8d3efaf92"
server_key_version: "encryptionkey@0"
I20260925 11:11:09.349210 25551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:09.385620 25551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:09.390293 25551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:09.390672 25551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:09.391815 25551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:09.393975 25613 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 11:11:09.401036 25551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 11:11:09.401225 25551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260925 11:11:09.401418 25551 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 11:11:09.404893 25551 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 11:11:09.405017 25551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260925 11:11:09.405634 25613 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Bootstrap starting.
I20260925 11:11:09.433060 25613 log.cc:824] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:09.450678 25613 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 11:11:09.451373 25613 tablet_bootstrap.cc:492] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Bootstrap complete.
I20260925 11:11:09.453032 25613 ts_tablet_manager.cc:1397] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Time spent bootstrapping tablet: real 0.048s user 0.036s sys 0.008s
I20260925 11:11:09.463936 25551 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:33589
I20260925 11:11:09.463954 25720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:33589 every 8 connection(s)
I20260925 11:11:09.464504 25613 raft_consensus.cc:348] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:09.465420 25551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:09.465487 25613 raft_consensus.cc:729] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106bcf16ce3946a999d73b26187ad37a, State: Initialized, Role: FOLLOWER
I20260925 11:11:09.466326 25613 consensus_queue.cc:260] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:09.468459 25613 ts_tablet_manager.cc:1428] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260925 11:11:09.470420 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.468557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:34591)
0925 11:11:09.469351 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.469579 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":316,"threads_started":1}
W20260925 11:11:09.471346 25721 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:36663: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
I20260925 11:11:09.471874 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25551
I20260925 11:11:09.472329 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.471907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:59183)
0925 11:11:09.472101 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.472186 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 11:11:09.472668 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:46443
--local_ip_for_outbound_sockets=127.8.181.2
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=41733
--webserver_interface=127.8.181.2
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:11:09.473518 25724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.473043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:53483)
0925 11:11:09.473236 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.473340 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 11:11:09.473858 25721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 11:11:09.812258 25725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:09.812613 25725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:09.812779 25725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:09.820303 25725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:09.820488 25725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:09.820678 25725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:09.830721 25725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:46443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=41733
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:09.832159 25725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:09.833498 25725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:09.847576 25731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:09.847707 25730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:09.848021 25733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:09.849164 25725 server_base.cc:1034] running on GCE node
I20260925 11:11:09.849898 25725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:09.851085 25725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:09.852329 25725 hybrid_clock.cc:648] HybridClock initialized: now 1790334669852274 us; error 51 us; skew 500 ppm
I20260925 11:11:09.852695 25725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:09 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334669, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:09.856420 25725 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:11:09.858153 25725 webserver.cc:460] Webserver started at http://127.8.181.2:41733/ using document root <none> and password file <none>
I20260925 11:11:09.859157 25725 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:09.859328 25725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:09.865278 25725 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:11:09.868664 25740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:09.870251 25725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:09.870458 25725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "259be75582e74ec582a119144f4e42c0"
format_stamp: "Formatted at 2026-09-25 11:10:23 on dist-test-slave-6042"
server_key: "eca5b99d37191d40f2cb939f2f9832d7"
server_key_iv: "6b32821f1ffd18653e9a44d0e93d0d24"
server_key_version: "encryptionkey@0"
I20260925 11:11:09.871580 25725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:09.886247 25725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:09.890767 25725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:09.891076 25725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:09.892195 25725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:09.893592 25725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:09.893723 25725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:09.893860 25725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:09.893950 25725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:09.929342 25725 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:46443
I20260925 11:11:09.929430 25852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:46443 every 8 connection(s)
I20260925 11:11:09.930902 25725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:09.935886 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.933926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:48841)
0925 11:11:09.934739 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.934955 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
W20260925 11:11:09.936635 25853 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:36663: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
I20260925 11:11:09.937677 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.937327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:60845)
0925 11:11:09.937484 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.937564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 11:11:09.938710 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:09.938326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:34433)
0925 11:11:09.938512 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:09.938600 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 11:11:09.938990 25853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 11:11:09.939076 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25725
I20260925 11:11:09.939750 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:44023
--local_ip_for_outbound_sockets=127.8.181.3
--tserver_master_addrs=127.8.181.62:36663
--webserver_port=33431
--webserver_interface=127.8.181.3
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:10.271611 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:10.271896 25856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:10.271991 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:10.279440 25856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:10.279565 25856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:10.279680 25856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:10.289417 25856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=33431
--tserver_master_addrs=127.8.181.62:36663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:10.290681 25856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:10.291957 25856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:10.306342 25862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:10.306612 25861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:10.307158 25864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:10.307699 25856 server_base.cc:1034] running on GCE node
I20260925 11:11:10.308429 25856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:10.309499 25856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:10.310755 25856 hybrid_clock.cc:648] HybridClock initialized: now 1790334670310713 us; error 38 us; skew 500 ppm
I20260925 11:11:10.311172 25856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:10 dist-test-slave-6042 krb5kdc[24401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334670, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:10.314582 25856 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:11:10.316186 25856 webserver.cc:460] Webserver started at http://127.8.181.3:33431/ using document root <none> and password file <none>
I20260925 11:11:10.316920 25856 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:10.317070 25856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:10.322613 25856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:10.325394 25871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:10.326777 25856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:10.326960 25856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "96af50b917a74c37b42de23cc8babb0e"
format_stamp: "Formatted at 2026-09-25 11:10:24 on dist-test-slave-6042"
server_key: "0fecb597ea5bda4869d80fee27945bd3"
server_key_iv: "529c5fa0b6d0f68127ea999e379edc86"
server_key_version: "encryptionkey@0"
I20260925 11:11:10.327950 25856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:10.354682 25856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:10.358933 25856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:10.359238 25856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:10.360338 25856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:10.361765 25856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:10.361897 25856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:10.362052 25856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:10.362145 25856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:10.397214 25856 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:44023
I20260925 11:11:10.397305 25983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:44023 every 8 connection(s)
I20260925 11:11:10.398790 25856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:10.403951 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.401814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:53271)
0925 11:11:10.402656 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.402993 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":307,"threads_started":1}
W20260925 11:11:10.404680 25984 heartbeater.cc:644] Failed to heartbeat to 127.8.181.62:36663 (0 consecutive failures): Network error: Failed to ping master at 127.8.181.62:36663: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
I20260925 11:11:10.405496 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 25856
I20260925 11:11:10.405726 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.405276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:57385)
0925 11:11:10.405468 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.405569 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 11:11:10.406697 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.406274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:43487)
0925 11:11:10.406456 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.406589 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 11:11:10.406960 25984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 11:11:10.475696 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.474573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:43709)
0925 11:11:10.475205 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.475391 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":360,"threads_started":1}
I20260925 11:11:10.839488 25989 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:10.839972 25989 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:10.841605 25989 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
Sep 25 11:11:10 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334669, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
Sep 25 11:11:10 dist-test-slave-6042 krb5kdc[24401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334669, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:11:10.865132 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.842340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:34717 (local address 127.8.181.3:44023)
0925 11:11:10.843304 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:10.843320 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:10.843776 (+ 456us) server_negotiation.cc:407] Connection header received
0925 11:11:10.845457 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:10.845465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:10.845796 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:10.846145 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:10.848102 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.849240 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.851449 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.852164 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.852450 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:10.856802 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:10.856815 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:10.856842 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:10.856897 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:10.860909 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:10.862374 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:10.862385 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:10.862395 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:10.862636 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:10.863032 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:10.863036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:10.863039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:10.863514 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:10.863709 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:10.864285 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 11:11:10.864706 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":305,"threads_started":1}
I20260925 11:11:10.865136 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.842253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:58529 (local address 127.8.181.2:46443)
0925 11:11:10.843008 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:10.843025 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:10.843222 (+ 197us) server_negotiation.cc:407] Connection header received
0925 11:11:10.845402 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:10.845412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:10.845804 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:10.846145 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:10.848046 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.849211 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.851455 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.852088 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.852300 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:10.856591 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:10.856605 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:10.856627 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:10.856686 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:10.860900 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:10.861819 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:10.861830 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:10.861922 (+ 92us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:10.862161 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:10.862715 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:10.862720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:10.862723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:10.863278 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:10.863510 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:10.864261 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 11:11:10.864678 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":421,"threads_started":1}
I20260925 11:11:10.865135 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.841976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:46443 (local address 127.8.181.1:58529)
0925 11:11:10.843009 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.843062 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:10.843693 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:10.846287 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:10.846299 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:10.847128 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:10.847837 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:10.847858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.849421 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.849425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:10.851300 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:10.851314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.852293 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.852299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:10.852588 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:10.853557 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:10.853584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:10.856231 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:10.861083 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:10.861090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:10.861105 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:10.861663 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:10.862293 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:10.862301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:10.862306 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:10.862584 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:10.863459 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:10.863467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:10.863763 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 11:11:10.864029 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 11:11:10.864654 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":131,"spinlock_wait_cycles":896}
I20260925 11:11:10.865296 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.841990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:44023 (local address 127.8.181.1:34717)
0925 11:11:10.843660 (+ 1670us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.843674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:10.843863 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:10.846287 (+ 2424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:10.846299 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:10.847128 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:10.847922 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:10.847936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.849409 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.849414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:10.851301 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:10.851314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:10.852445 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:10.852451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:10.852611 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:10.853530 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:10.853559 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:10.856508 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:10.861778 (+ 5270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:10.861781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:10.861783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:10.862207 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:10.862757 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:10.862762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:10.862766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:10.862907 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:10.863646 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:10.863651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:10.863829 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 11:11:10.864029 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 11:11:10.864847 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"mutex_wait_us":954,"spinlock_wait_cycles":41216,"thread_start_us":867,"threads_started":1}
W20260925 11:11:10.868618 25609 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
W20260925 11:11:10.868883 25608 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
I20260925 11:11:10.869045 25608 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a; no voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e
I20260925 11:11:10.869495 25989 raft_consensus.cc:2726] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:11:10.940690 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:10.939620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:50853)
0925 11:11:10.940159 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:10.940353 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
I20260925 11:11:11.408352 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:11.407504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:50351)
0925 11:11:11.408022 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:11.408145 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260925 11:11:11.477562 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:11.476862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:45751)
0925 11:11:11.477306 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:11.477399 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260925 11:11:11.942376 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:11.941568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:45575)
0925 11:11:11.942122 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:11.942208 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":322,"threads_started":1}
I20260925 11:11:12.410156 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:12.409358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:55023)
0925 11:11:12.409851 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:12.409951 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260925 11:11:12.479748 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:12.478925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:54237)
0925 11:11:12.479438 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:12.479532 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":247,"threads_started":1}
I20260925 11:11:12.662351 25999 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:12.662642 25999 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:12.663443 25999 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
W20260925 11:11:12.664674 25609 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
W20260925 11:11:12.664893 25608 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
I20260925 11:11:12.664999 25608 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a; no voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e
I20260925 11:11:12.665228 25999 raft_consensus.cc:2726] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:11:12.944080 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:12.943388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:48499)
0925 11:11:12.943827 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:12.943906 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260925 11:11:13.412017 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:13.411240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:33455)
0925 11:11:13.411722 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:13.411822 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20260925 11:11:13.481642 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:13.480867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:33339)
0925 11:11:13.481355 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:13.481466 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":267,"threads_started":1}
I20260925 11:11:13.945698 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:13.944996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:38639)
0925 11:11:13.945424 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:13.945513 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260925 11:11:14.182888 26004 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:14.183082 26004 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:14.183807 26004 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
W20260925 11:11:14.184839 25608 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
W20260925 11:11:14.185050 25609 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
I20260925 11:11:14.185178 25609 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a; no voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e
I20260925 11:11:14.185493 26004 raft_consensus.cc:2726] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:11:14.413822 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:14.413048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:34703)
0925 11:11:14.413543 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:14.413641 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":251,"threads_started":1}
I20260925 11:11:14.483734 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:14.482934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:53583)
0925 11:11:14.483449 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:14.483545 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":265,"threads_started":1}
I20260925 11:11:14.947556 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:14.946777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:53729)
0925 11:11:14.947258 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:14.947350 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":263,"threads_started":1}
I20260925 11:11:15.415630 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:15.414858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:50381)
0925 11:11:15.415376 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:15.415466 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":273,"threads_started":1}
I20260925 11:11:15.485709 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:15.484860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:34307)
0925 11:11:15.485365 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:15.485495 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":293,"threads_started":1}
I20260925 11:11:15.949445 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:15.948624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:40819)
0925 11:11:15.949178 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:15.949264 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":279,"threads_started":1}
I20260925 11:11:16.417279 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:16.416523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:56491)
0925 11:11:16.416980 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:16.417097 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":250,"threads_started":1}
I20260925 11:11:16.487557 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:16.486814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:59641)
0925 11:11:16.487273 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:16.487358 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260925 11:11:16.951524 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:16.950661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:45017)
0925 11:11:16.951231 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:16.951325 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
I20260925 11:11:17.419100 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:17.418286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:52195)
0925 11:11:17.418803 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:17.418888 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":262,"threads_started":1}
I20260925 11:11:17.489323 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:17.488538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:44365)
0925 11:11:17.489016 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:17.489149 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260925 11:11:17.953660 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:17.952654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:34523)
0925 11:11:17.953352 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:17.953460 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":475,"threads_started":1}
I20260925 11:11:18.421103 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:18.420252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:47181)
0925 11:11:18.420801 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:18.420892 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":295,"threads_started":1}
I20260925 11:11:18.491438 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:18.490419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:47133)
0925 11:11:18.491158 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:18.491242 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":342,"threads_started":1}
I20260925 11:11:18.955654 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:18.954850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:32807)
0925 11:11:18.955390 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:18.955490 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260925 11:11:19.166980 26020 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:19.167215 26020 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:19.167867 26020 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
W20260925 11:11:19.169041 25608 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
W20260925 11:11:19.169301 25609 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
I20260925 11:11:19.169423 25609 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a; no voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e
I20260925 11:11:19.169767 26020 raft_consensus.cc:2726] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:11:19.423125 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:19.422203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:46293)
0925 11:11:19.422763 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:19.422963 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
I20260925 11:11:19.493789 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:19.492709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:48125)
0925 11:11:19.493376 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:19.493510 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":349,"threads_started":1}
I20260925 11:11:19.957700 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:19.956848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:51481)
0925 11:11:19.957418 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:19.957507 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":348,"threads_started":1}
I20260925 11:11:20.424990 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:20.424148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:36555)
0925 11:11:20.424645 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:20.424737 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260925 11:11:20.495671 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:20.494921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:52605)
0925 11:11:20.495392 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:20.495494 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
I20260925 11:11:20.959621 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:20.958795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:33369)
0925 11:11:20.959349 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:20.959439 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":237,"threads_started":1}
I20260925 11:11:21.426802 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:21.426072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:57165)
0925 11:11:21.426555 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:21.426642 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":230,"threads_started":1}
I20260925 11:11:21.497540 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:21.496705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:49281)
0925 11:11:21.497266 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:21.497356 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":336,"threads_started":1}
I20260925 11:11:21.961437 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:21.960681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:45567)
0925 11:11:21.961171 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:21.961257 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":233,"threads_started":1}
I20260925 11:11:22.428622 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:22.427862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:39897)
0925 11:11:22.428364 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:22.428450 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":282,"threads_started":1}
I20260925 11:11:22.499794 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:22.498901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:48607)
0925 11:11:22.499528 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:22.499611 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":317,"threads_started":1}
I20260925 11:11:22.963416 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:22.962600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:42207)
0925 11:11:22.963124 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:22.963221 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260925 11:11:23.430374 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:23.429597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:53163)
0925 11:11:23.430116 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:23.430205 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
I20260925 11:11:23.501632 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:23.500862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:43219)
0925 11:11:23.501352 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:23.501446 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260925 11:11:23.965214 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:23.964460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:57255)
0925 11:11:23.964924 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:23.965015 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260925 11:11:24.432410 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:24.431568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:60301)
0925 11:11:24.432047 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:24.432137 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260925 11:11:24.504196 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:24.503185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:50581)
0925 11:11:24.503847 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:24.503948 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
I20260925 11:11:24.967429 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:24.966560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.2:36173)
0925 11:11:24.967089 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:24.967246 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":286,"threads_started":1}
I20260925 11:11:25.343878 26039 raft_consensus.cc:482] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:25.344071 26039 raft_consensus.cc:504] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "106bcf16ce3946a999d73b26187ad37a" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 33589 } } peers { permanent_uuid: "259be75582e74ec582a119144f4e42c0" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 46443 } } peers { permanent_uuid: "96af50b917a74c37b42de23cc8babb0e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 44023 } }
I20260925 11:11:25.344807 26039 leader_election.cc:288] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443), 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023)
W20260925 11:11:25.345821 25609 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96af50b917a74c37b42de23cc8babb0e (127.8.181.3:44023): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
W20260925 11:11:25.346042 25608 leader_election.cc:341] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 259be75582e74ec582a119144f4e42c0 (127.8.181.2:46443): Not found: Tablet not found: 6701b05d1914438db2c91ceee76e87f3
I20260925 11:11:25.346225 25608 leader_election.cc:302] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 106bcf16ce3946a999d73b26187ad37a; no voters: 259be75582e74ec582a119144f4e42c0, 96af50b917a74c37b42de23cc8babb0e
I20260925 11:11:25.346531 26039 raft_consensus.cc:2726] T 6701b05d1914438db2c91ceee76e87f3 P 106bcf16ce3946a999d73b26187ad37a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 11:11:25.408944 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25551
I20260925 11:11:25.425485 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:25.410084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.8.181.1:33589)
0925 11:11:25.411165 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:25.411175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:25.411212 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:11:25.411303 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:25.411315 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:25.411645 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:25.411823 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:25.413255 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.414269 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.415596 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.416189 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.416287 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:25.419114 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:25.419125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:25.419139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:25.419168 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:25.422524 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.423312 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.423319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.423321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.423447 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.423844 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.423848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.423850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.424284 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:25.424553 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:25.424807 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 11:11:25.425141 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":657,"threads_started":1}
I20260925 11:11:25.427089 25716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:25.434396 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:25.433604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.3:38409)
0925 11:11:25.434138 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:25.434246 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":263,"threads_started":1}
I20260925 11:11:25.519979 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:25.518819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:36663 (local address 127.8.181.1:55505)
0925 11:11:25.519416 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:25.519643 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.62:36663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":302,"threads_started":1}
I20260925 11:11:25.520426 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25551
I20260925 11:11:25.540191 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25725
I20260925 11:11:25.554298 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:25.541086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.8.181.2:46443)
0925 11:11:25.541592 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:25.541598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:25.541615 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:11:25.541671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:25.541675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:25.541732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:25.541835 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:25.543323 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.544154 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.545429 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.545861 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.545959 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:25.548737 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:25.548742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:25.548745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:25.548781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:25.551734 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.552454 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.552458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.552460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.552602 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.552943 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.552946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.552948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.553208 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:25.553411 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:25.553692 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 11:11:25.553868 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260925 11:11:25.555230 25848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:25.641129 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25725
I20260925 11:11:25.658403 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 25856
I20260925 11:11:25.673134 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:25.659311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.8.181.3:44023)
0925 11:11:25.659945 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:25.659950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:25.659967 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:11:25.660021 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:25.660025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:25.660099 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:25.660192 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:25.661801 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.662985 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.664241 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:25.664667 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:25.664762 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:25.667638 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:25.667644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:25.667647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:25.667699 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:25.670558 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.671300 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.671304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.671306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.671441 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:25.671807 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:25.671810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:25.671812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:25.672050 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:25.672248 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:25.672521 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 11:11:25.672730 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":376,"threads_started":1}
I20260925 11:11:25.673951 25979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:25.760771 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 25856
2026-09-25T11:11:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64734 ms)
[----------] 30 tests from SecurityITest (309398 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:25 dist-test-slave-6042 krb5kdc[26050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:25 dist-test-slave-6042 krb5kdc[26050](info): set up 2 sockets
Sep 25 11:11:25 dist-test-slave-6042 krb5kdc[26050](info): commencing operation
krb5kdc: starting...
W20260925 11:11:26.887027 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:27 dist-test-slave-6042 krb5kdc[26050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:11:27.159138 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41543
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:27.492190 26063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:27.492514 26063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:27.492606 26063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:27.499928 26063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:27.500054 26063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:27.500101 26063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:27.500151 26063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:27.500196 26063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:27.510358 26063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:27.511749 26063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:27.512997 26063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:27.526104 26069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:27.526206 26071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:27.526187 26068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:27.531945 26063 server_base.cc:1034] running on GCE node
I20260925 11:11:27.532822 26063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:27.534636 26063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:27.535863 26063 hybrid_clock.cc:648] HybridClock initialized: now 1790334687535785 us; error 73 us; skew 500 ppm
I20260925 11:11:27.536191 26063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:27 dist-test-slave-6042 krb5kdc[26050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:27.539590 26063 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:11:27.541326 26063 webserver.cc:460] Webserver started at http://127.8.181.62:39719/ using document root <none> and password file <none>
I20260925 11:11:27.542093 26063 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:27.542229 26063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:27.542639 26063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:27.545910 26063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "a02105a3283d45b1ad73c9a11e15c9a0"
format_stamp: "Formatted at 2026-09-25 11:11:27 on dist-test-slave-6042"
server_key: "6795f2af7266be29913663ba1e6314aa"
server_key_iv: "b0c585316cd4270fd8c1c6d4cee3a95a"
server_key_version: "encryptionkey@0"
I20260925 11:11:27.546799 26063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "a02105a3283d45b1ad73c9a11e15c9a0"
format_stamp: "Formatted at 2026-09-25 11:11:27 on dist-test-slave-6042"
server_key: "6795f2af7266be29913663ba1e6314aa"
server_key_iv: "b0c585316cd4270fd8c1c6d4cee3a95a"
server_key_version: "encryptionkey@0"
I20260925 11:11:27.552551 26063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:11:27.556548 26078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:27.558056 26063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:27.558270 26063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a02105a3283d45b1ad73c9a11e15c9a0"
format_stamp: "Formatted at 2026-09-25 11:11:27 on dist-test-slave-6042"
server_key: "6795f2af7266be29913663ba1e6314aa"
server_key_iv: "b0c585316cd4270fd8c1c6d4cee3a95a"
server_key_version: "encryptionkey@0"
I20260925 11:11:27.558547 26063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:27.572619 26063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:27.577732 26063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:27.578086 26063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:27.599500 26063 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:41543
I20260925 11:11:27.599541 26129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:41543 every 8 connection(s)
I20260925 11:11:27.601154 26063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:27.605044 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26063
I20260925 11:11:27.605351 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:27.605779 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:27.606289 8916 external_mini_cluster.cc:1428] Setting key 4dbfd885584c9403bb1c499034493e80
I20260925 11:11:27.622088 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: Bootstrap starting.
Sep 25 11:11:27 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:27.625821 26130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:27.627111 26130 log.cc:824] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:27.630157 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: No bootstrap required, opened a new log
I20260925 11:11:27.634181 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:27.608654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.8.181.62:41543)
0925 11:11:27.609662 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:27.609675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:27.609723 (+ 48us) server_negotiation.cc:407] Connection header received
0925 11:11:27.611634 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:27.611660 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:27.612172 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:27.612581 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:27.614800 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:27.616333 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:27.618423 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:27.619354 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:27.619526 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:27.624796 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:27.624806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:27.624824 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:27.624879 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:27.628789 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:27.629703 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:27.629710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:27.629717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:27.629922 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:27.630397 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:27.630402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:27.630405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:27.631261 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:27.631562 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:27.632170 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 11:11:27.632672 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":503,"threads_started":1}
I20260925 11:11:27.639274 26130 raft_consensus.cc:348] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } }
I20260925 11:11:27.639657 26130 raft_consensus.cc:374] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:27.639734 26130 raft_consensus.cc:729] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a02105a3283d45b1ad73c9a11e15c9a0, State: Initialized, Role: FOLLOWER
I20260925 11:11:27.640198 26130 consensus_queue.cc:260] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } }
I20260925 11:11:27.640385 26130 raft_consensus.cc:388] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:27.640488 26130 raft_consensus.cc:482] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:27.640612 26130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:27.643101 26130 raft_consensus.cc:504] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } }
I20260925 11:11:27.643671 26130 leader_election.cc:302] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a02105a3283d45b1ad73c9a11e15c9a0; no voters:
I20260925 11:11:27.644469 26130 leader_election.cc:288] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:27.644654 26135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:27.645449 26135 raft_consensus.cc:686] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [term 1 LEADER]: Becoming Leader. State: Replica: a02105a3283d45b1ad73c9a11e15c9a0, State: Running, Role: LEADER
I20260925 11:11:27.646026 26135 consensus_queue.cc:237] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } }
I20260925 11:11:27.646781 26130 sys_catalog.cc:564] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:27.649138 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a02105a3283d45b1ad73c9a11e15c9a0. Latest consensus state: current_term: 1 leader_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } } }
I20260925 11:11:27.650089 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:27.649622 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a02105a3283d45b1ad73c9a11e15c9a0" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41543 } } }
I20260925 11:11:27.650568 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:27.651603 26141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:27.656775 26141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:27.666203 26141 catalog_manager.cc:1347] Generated new cluster ID: f24ef6d4d25043809661ccb233d4f02f
I20260925 11:11:27.666338 26141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:27.695787 26141 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:27.697777 26141 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:27.705957 26141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: Generated new TSK 0
I20260925 11:11:27.707013 26141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:11:27.791064 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:28.152562 26156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:28.152906 26156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:28.153088 26156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:28.161444 26156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:28.161558 26156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:28.161726 26156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:28.172197 26156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:28.173570 26156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:28.174966 26156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:28.188323 26161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:28.188424 26164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:28.188455 26162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:28.189298 26156 server_base.cc:1034] running on GCE node
I20260925 11:11:28.189944 26156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:28.191097 26156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:28.192338 26156 hybrid_clock.cc:648] HybridClock initialized: now 1790334688192290 us; error 42 us; skew 500 ppm
I20260925 11:11:28.192677 26156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:28 dist-test-slave-6042 krb5kdc[26050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334688, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:28.196033 26156 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:11:28.197577 26156 webserver.cc:460] Webserver started at http://127.8.181.1:43709/ using document root <none> and password file <none>
I20260925 11:11:28.198256 26156 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:28.198360 26156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:28.198732 26156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:28.201860 26156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "5da58acd969f401c97c1086bd8fdc4b6"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "0c5a9f6d7a590587c3c818956b74cf63"
server_key_iv: "1a7ef41fcf8644c9964127e6a5c07ec1"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.202800 26156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "5da58acd969f401c97c1086bd8fdc4b6"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "0c5a9f6d7a590587c3c818956b74cf63"
server_key_iv: "1a7ef41fcf8644c9964127e6a5c07ec1"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.208679 26156 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:11:28.213370 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.215061 26156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:28.215250 26156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "5da58acd969f401c97c1086bd8fdc4b6"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "0c5a9f6d7a590587c3c818956b74cf63"
server_key_iv: "1a7ef41fcf8644c9964127e6a5c07ec1"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.215484 26156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:28.244958 26156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:28.249842 26156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:28.250178 26156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:28.251283 26156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:28.252756 26156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:28.252832 26156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.253015 26156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:28.253085 26156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.288023 26156 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:37893
I20260925 11:11:28.288072 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:37893 every 8 connection(s)
I20260925 11:11:28.289379 26156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:28.293727 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26156
I20260925 11:11:28.293941 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:28.294682 8916 external_mini_cluster.cc:1428] Setting key 2670b54750732fade9e232bf415ee549
Sep 25 11:11:28 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334688, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:28.311331 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:28.292115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:54353 (local address 127.8.181.62:41543)
0925 11:11:28.292587 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:28.292591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:28.293579 (+ 988us) server_negotiation.cc:407] Connection header received
0925 11:11:28.295804 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:28.295810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:28.295892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:28.296007 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:28.297621 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.298359 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.299961 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.300395 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.300554 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:28.304188 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:28.304214 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:28.304222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:28.304274 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:28.307827 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:28.308626 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:28.308635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:28.308641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:28.308803 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:28.309220 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:28.309227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:28.309233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:28.309635 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:28.309801 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:28.310561 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 11:11:28.310826 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260925 11:11:28.312011 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:28.292654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41543 (local address 127.8.181.1:54353)
0925 11:11:28.293372 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:28.293452 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:28.295491 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:28.296207 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:28.296216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:28.296863 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:28.297447 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:28.297466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.298549 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.298553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:28.299800 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:28.299813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.300638 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.300647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:28.300908 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:28.301623 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:28.301647 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:28.303948 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:28.308000 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:28.308008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:28.308022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:28.308480 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:28.308914 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:28.308917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:28.308919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:28.309094 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:28.309770 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:28.309777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:28.310124 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 11:11:28.310353 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 11:11:28.310852 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":294,"threads_started":1}
I20260925 11:11:28.313716 26284 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41543
I20260925 11:11:28.314067 26284 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:28.315011 26284 heartbeater.cc:511] Master 127.8.181.62:41543 requested a full tablet report, sending...
I20260925 11:11:28.317381 26095 ts_manager.cc:194] Registered new tserver with Master: 5da58acd969f401c97c1086bd8fdc4b6 (127.8.181.1:37893)
I20260925 11:11:28.319005 26095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:54353
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:11:28.358429 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:28.705950 26290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:28.706300 26290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:28.706518 26290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:28.714390 26290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:28.714576 26290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:28.714751 26290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:28.725171 26290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:28.726616 26290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:28.727978 26290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:28.741456 26296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:28.741480 26295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:28.741854 26298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:28.742290 26290 server_base.cc:1034] running on GCE node
I20260925 11:11:28.742961 26290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:28.744031 26290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:28.745263 26290 hybrid_clock.cc:648] HybridClock initialized: now 1790334688745219 us; error 37 us; skew 500 ppm
I20260925 11:11:28.745602 26290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:28 dist-test-slave-6042 krb5kdc[26050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334688, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:28.749244 26290 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:11:28.750799 26290 webserver.cc:460] Webserver started at http://127.8.181.2:42289/ using document root <none> and password file <none>
I20260925 11:11:28.751544 26290 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:28.751677 26290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:28.752005 26290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:28.755193 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "02ed89df45ac438c842c68147b3ba311"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "34813e360417b486a5c9a221657ad5b0"
server_key_iv: "7df51a95e9a3cd78447558b22a6ae64a"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.756014 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "02ed89df45ac438c842c68147b3ba311"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "34813e360417b486a5c9a221657ad5b0"
server_key_iv: "7df51a95e9a3cd78447558b22a6ae64a"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.761340 26290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 11:11:28.765116 26305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.766510 26290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:28.766695 26290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "02ed89df45ac438c842c68147b3ba311"
format_stamp: "Formatted at 2026-09-25 11:11:28 on dist-test-slave-6042"
server_key: "34813e360417b486a5c9a221657ad5b0"
server_key_iv: "7df51a95e9a3cd78447558b22a6ae64a"
server_key_version: "encryptionkey@0"
I20260925 11:11:28.766911 26290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:28.783360 26290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:28.787894 26290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:28.788211 26290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:28.789213 26290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:28.790680 26290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:28.790784 26290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.790953 26290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:28.791033 26290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:28.825098 26290 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:45277
I20260925 11:11:28.825208 26417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:45277 every 8 connection(s)
I20260925 11:11:28.826422 26290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:28.835665 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26290
I20260925 11:11:28.835865 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:28.836428 8916 external_mini_cluster.cc:1428] Setting key 1eab141c2e3d9eac8fe3880b4f50ff9a
Sep 25 11:11:28 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334688, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:28.848132 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:28.828960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:52801 (local address 127.8.181.62:41543)
0925 11:11:28.829392 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:28.829399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:28.830351 (+ 952us) server_negotiation.cc:407] Connection header received
0925 11:11:28.832206 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:28.832211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:28.832280 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:28.832380 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:28.833805 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.834593 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.836211 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.837009 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.837156 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:28.840901 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:28.840910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:28.840918 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:28.840965 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:28.844619 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:28.845386 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:28.845394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:28.845400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:28.845567 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:28.845991 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:28.845998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:28.846004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:28.846399 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:28.846603 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:28.847479 (+ 876us) server_negotiation.cc:299] Negotiation successful
0925 11:11:28.847744 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260925 11:11:28.849412 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:28.829520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41543 (local address 127.8.181.2:52801)
0925 11:11:28.830197 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:28.830234 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:28.831953 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:28.832596 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:28.832605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:28.833123 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:28.833654 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:28.833670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.834728 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.834732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:28.836075 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:28.836086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:28.837281 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:28.837289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:28.837537 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:28.838292 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:28.838326 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:28.840654 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:28.844780 (+ 4126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:28.844787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:28.844801 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:28.845257 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:28.845679 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:28.845683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:28.845686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:28.845874 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:28.846587 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:28.846593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:28.847042 (+ 449us) client_negotiation.cc:769] Sending connection context
0925 11:11:28.847301 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 11:11:28.847873 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":275,"threads_started":1}
I20260925 11:11:28.851207 26418 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41543
I20260925 11:11:28.851567 26418 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:28.852459 26418 heartbeater.cc:511] Master 127.8.181.62:41543 requested a full tablet report, sending...
I20260925 11:11:28.854262 26095 ts_manager.cc:194] Registered new tserver with Master: 02ed89df45ac438c842c68147b3ba311 (127.8.181.2:45277)
I20260925 11:11:28.855597 26095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:52801
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:11:28.898731 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:29.236652 26424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:29.237005 26424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:29.237181 26424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:29.244513 26424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:29.244685 26424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:29.244864 26424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:29.255154 26424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:29.256494 26424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:29.257825 26424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:29.271279 26430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:29.271384 26429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:29.271346 26432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:29.272228 26424 server_base.cc:1034] running on GCE node
I20260925 11:11:29.272877 26424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:29.273957 26424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:29.275197 26424 hybrid_clock.cc:648] HybridClock initialized: now 1790334689275140 us; error 51 us; skew 500 ppm
I20260925 11:11:29.275561 26424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:29 dist-test-slave-6042 krb5kdc[26050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:29.278828 26424 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:11:29.280362 26424 webserver.cc:460] Webserver started at http://127.8.181.3:35131/ using document root <none> and password file <none>
I20260925 11:11:29.281018 26424 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:29.281128 26424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:29.281387 26424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:29.284327 26424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "3d3f5258368a44abb189fe58443fee5f"
format_stamp: "Formatted at 2026-09-25 11:11:29 on dist-test-slave-6042"
server_key: "99eb522ecf07e1ba20639af6b0dae9eb"
server_key_iv: "fa33b68a9e2144e18312adae8ee7dc36"
server_key_version: "encryptionkey@0"
I20260925 11:11:29.285048 26424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "3d3f5258368a44abb189fe58443fee5f"
format_stamp: "Formatted at 2026-09-25 11:11:29 on dist-test-slave-6042"
server_key: "99eb522ecf07e1ba20639af6b0dae9eb"
server_key_iv: "fa33b68a9e2144e18312adae8ee7dc36"
server_key_version: "encryptionkey@0"
I20260925 11:11:29.291164 26424 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:11:29.295588 26439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:29.297132 26424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:29.297309 26424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "3d3f5258368a44abb189fe58443fee5f"
format_stamp: "Formatted at 2026-09-25 11:11:29 on dist-test-slave-6042"
server_key: "99eb522ecf07e1ba20639af6b0dae9eb"
server_key_iv: "fa33b68a9e2144e18312adae8ee7dc36"
server_key_version: "encryptionkey@0"
I20260925 11:11:29.297513 26424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:29.322861 26284 heartbeater.cc:503] Master 127.8.181.62:41543 was elected leader, sending a full tablet report...
I20260925 11:11:29.323359 26424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:29.328007 26424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:29.328318 26424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:29.329308 26424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:29.330739 26424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:29.330859 26424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:29.331023 26424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:29.331103 26424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:29.366921 26424 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:36513
I20260925 11:11:29.366976 26551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:36513 every 8 connection(s)
I20260925 11:11:29.368281 26424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:29.375123 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26424
I20260925 11:11:29.375339 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:29.375814 8916 external_mini_cluster.cc:1428] Setting key b3c17804e52dcb900a49b0dc9af0c3c1
Sep 25 11:11:29 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:29.389837 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.371000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:56827 (local address 127.8.181.62:41543)
0925 11:11:29.371509 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.371516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.372664 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 11:11:29.374571 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.374575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.374647 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.374766 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:29.376465 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.377447 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.379101 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.379510 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.379644 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.383324 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:29.383329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:29.383332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:29.383366 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:29.386293 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:29.387259 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:29.387273 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:29.387277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:29.387407 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:29.387889 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:29.387894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:29.387896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:29.388192 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:29.388371 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.389153 (+ 782us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.389406 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":201,"threads_started":1}
I20260925 11:11:29.391111 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.371629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41543 (local address 127.8.181.3:56827)
0925 11:11:29.372498 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.372540 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.374258 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.374964 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.374972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.375606 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:29.376274 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.376293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.377612 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.377616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.378946 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.378958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.379747 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.379753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.379983 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.380715 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:29.380738 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:29.383098 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:29.386517 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:29.386527 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:29.386547 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:29.387109 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:29.387531 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:29.387537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:29.387541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:29.387748 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:29.388339 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:29.388349 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:29.388737 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.389029 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.389642 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":363,"threads_started":1}
I20260925 11:11:29.392532 26552 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41543
I20260925 11:11:29.392817 26552 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:29.393482 26552 heartbeater.cc:511] Master 127.8.181.62:41543 requested a full tablet report, sending...
I20260925 11:11:29.395037 26095 ts_manager.cc:194] Registered new tserver with Master: 3d3f5258368a44abb189fe58443fee5f (127.8.181.3:36513)
I20260925 11:11:29.396040 26095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:56827
I20260925 11:11:29.403978 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:29.421984 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.408619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.8.181.62:41543)
0925 11:11:29.408788 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.408793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.409279 (+ 486us) server_negotiation.cc:407] Connection header received
0925 11:11:29.409423 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.409427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.409491 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.409598 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:29.411234 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.411952 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.413374 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.413761 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.413855 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.416480 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:29.416485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:29.416488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:29.416519 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:29.419192 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:29.419946 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:29.419953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:29.419956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:29.420079 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:29.420470 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:29.420476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:29.420480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:29.420823 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:29.421092 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.421427 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.421665 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 11:11:29.426257 26095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:29.433305 26095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:29.461927 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.446415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.8.181.3:36513)
0925 11:11:29.453793 (+ 7378us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.453802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.453837 (+ 35us) server_negotiation.cc:407] Connection header received
0925 11:11:29.453998 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.454006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.454350 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.454603 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:29.455303 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.456857 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.459595 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.460693 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.460825 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.461085 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.461374 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.461544 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7187,"thread_start_us":7010,"threads_started":1}
I20260925 11:11:29.461935 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.446163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:36513 (local address 127.0.0.1:49302)
0925 11:11:29.447877 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.447918 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.448123 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.454614 (+ 6491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.454622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.454652 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:29.455055 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.455078 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.457004 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.457014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.459437 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.459448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.460902 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.460909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.461123 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.461190 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.461329 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.461524 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":7727,"threads_started":1}
I20260925 11:11:29.463088 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.446939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45277 (local address 127.0.0.1:40208)
0925 11:11:29.454195 (+ 7256us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.454216 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.454339 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.455496 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.455504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.455539 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:29.456032 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.456047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.457858 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.457864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.460248 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.460262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.462231 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.462239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.462365 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.462387 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.462529 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.462673 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7096,"mutex_wait_us":24,"thread_start_us":766,"threads_started":1}
I20260925 11:11:29.463814 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.447557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.8.181.2:45277)
0925 11:11:29.454615 (+ 7058us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.454624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.454655 (+ 31us) server_negotiation.cc:407] Connection header received
0925 11:11:29.454759 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.454766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.455006 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.455246 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:29.456270 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.457686 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.460459 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.461957 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.462097 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.462284 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.462588 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.462747 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5640,"thread_start_us":5477,"threads_started":1}
I20260925 11:11:29.464918 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.454681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.8.181.1:37893)
0925 11:11:29.455634 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.455642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.456091 (+ 449us) server_negotiation.cc:407] Connection header received
0925 11:11:29.456261 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.456268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.456486 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.456656 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:29.457273 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.458872 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.461189 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.463126 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.463523 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.463766 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.464441 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.464575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":608,"threads_started":1}
I20260925 11:11:29.464938 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.455014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:37893 (local address 127.0.0.1:40074)
0925 11:11:29.455944 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.455967 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.456107 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.456675 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.456680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.456704 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:29.457069 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.457081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.459024 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.459029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.461007 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.461020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.464063 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.464072 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.464257 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.464287 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.464388 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.464544 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":1,"thread_start_us":643,"threads_started":1}
I20260925 11:11:29.469142 26353 tablet_service.cc:1467] Processing CreateTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c (DEFAULT_TABLE table=test-table [id=6663ed3028da4523bee8377050d42378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:29.470922 26353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56595ad3f8534ad6b7c19885142a8f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:29.472975 26219 tablet_service.cc:1467] Processing CreateTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c (DEFAULT_TABLE table=test-table [id=6663ed3028da4523bee8377050d42378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:29.474370 26219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56595ad3f8534ad6b7c19885142a8f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:29.466785 26487 tablet_service.cc:1467] Processing CreateTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c (DEFAULT_TABLE table=test-table [id=6663ed3028da4523bee8377050d42378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:29.476197 26487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56595ad3f8534ad6b7c19885142a8f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:29.487253 26571 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Bootstrap starting.
I20260925 11:11:29.490370 26572 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Bootstrap starting.
I20260925 11:11:29.491047 26571 tablet_bootstrap.cc:654] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:29.492340 26571 log.cc:824] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:29.493817 26572 tablet_bootstrap.cc:654] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:29.494912 26572 log.cc:824] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:29.495724 26571 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: No bootstrap required, opened a new log
I20260925 11:11:29.496016 26571 ts_tablet_manager.cc:1397] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 11:11:29.496039 26574 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Bootstrap starting.
I20260925 11:11:29.500232 26574 tablet_bootstrap.cc:654] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:29.501009 26572 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: No bootstrap required, opened a new log
I20260925 11:11:29.501297 26572 ts_tablet_manager.cc:1397] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260925 11:11:29.501477 26574 log.cc:824] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:29.504251 26574 tablet_bootstrap.cc:492] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: No bootstrap required, opened a new log
I20260925 11:11:29.504529 26574 ts_tablet_manager.cc:1397] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 11:11:29.506420 26571 raft_consensus.cc:348] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.506915 26571 raft_consensus.cc:374] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:29.507030 26571 raft_consensus.cc:729] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ed89df45ac438c842c68147b3ba311, State: Initialized, Role: FOLLOWER
I20260925 11:11:29.507742 26571 consensus_queue.cc:260] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.508962 26418 heartbeater.cc:503] Master 127.8.181.62:41543 was elected leader, sending a full tablet report...
I20260925 11:11:29.509586 26571 ts_tablet_manager.cc:1428] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260925 11:11:29.512015 26572 raft_consensus.cc:348] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.512528 26572 raft_consensus.cc:374] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:29.512641 26572 raft_consensus.cc:729] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3f5258368a44abb189fe58443fee5f, State: Initialized, Role: FOLLOWER
I20260925 11:11:29.513319 26572 consensus_queue.cc:260] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.514137 26574 raft_consensus.cc:348] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.514622 26574 raft_consensus.cc:374] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:29.514710 26574 raft_consensus.cc:729] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da58acd969f401c97c1086bd8fdc4b6, State: Initialized, Role: FOLLOWER
I20260925 11:11:29.514991 26552 heartbeater.cc:503] Master 127.8.181.62:41543 was elected leader, sending a full tablet report...
I20260925 11:11:29.515360 26574 consensus_queue.cc:260] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.515704 26572 ts_tablet_manager.cc:1428] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 11:11:29.517037 26574 ts_tablet_manager.cc:1428] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260925 11:11:29.542078 26285 tablet.cc:2369] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:11:29.578588 26419 tablet.cc:2369] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:11:29.619977 26553 tablet.cc:2369] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:29.893229 26579 raft_consensus.cc:482] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:29.893507 26579 raft_consensus.cc:504] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.895298 26579 leader_election.cc:288] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d3f5258368a44abb189fe58443fee5f (127.8.181.3:36513), 02ed89df45ac438c842c68147b3ba311 (127.8.181.2:45277)
I20260925 11:11:29.902051 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.895672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:56805 (local address 127.8.181.3:36513)
0925 11:11:29.895864 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.895870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.896604 (+ 734us) server_negotiation.cc:407] Connection header received
0925 11:11:29.896826 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.896831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.896898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.896964 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:29.897596 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.898328 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.900192 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.901009 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.901110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.901186 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.901660 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.901789 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 11:11:29.902436 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.895484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:36513 (local address 127.8.181.1:56805)
0925 11:11:29.896450 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.896471 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.896666 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.897010 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.897016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.897045 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:29.897437 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.897447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.898461 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.898491 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.900035 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.900047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.901216 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.901222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.901472 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.901507 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.901605 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.901788 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":58,"thread_start_us":529,"threads_started":1}
I20260925 11:11:29.902487 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.895846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:49169 (local address 127.8.181.2:45277)
0925 11:11:29.896074 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:29.896080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:29.896690 (+ 610us) server_negotiation.cc:407] Connection header received
0925 11:11:29.896874 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:29.896878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:29.896962 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:29.897058 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:29.897743 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.898549 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.900346 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.901289 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.901438 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.901527 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:29.901890 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 11:11:29.902012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:11:29.903010 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:29.895774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:45277 (local address 127.8.181.1:49169)
0925 11:11:29.896526 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:29.896574 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:29.896708 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:29.897107 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:29.897112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:29.897156 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:29.897551 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.897565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.898680 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.898684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.900142 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:29.900158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:29.901500 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:29.901505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:29.901709 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:29.901741 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 11:11:29.902500 (+ 759us) client_negotiation.cc:240] Negotiation successful
0925 11:11:29.902656 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":3,"thread_start_us":481,"threads_started":1}
I20260925 11:11:29.903682 26507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56595ad3f8534ad6b7c19885142a8f5c" candidate_uuid: "5da58acd969f401c97c1086bd8fdc4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3f5258368a44abb189fe58443fee5f" is_pre_election: true
I20260925 11:11:29.904047 26373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56595ad3f8534ad6b7c19885142a8f5c" candidate_uuid: "5da58acd969f401c97c1086bd8fdc4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ed89df45ac438c842c68147b3ba311" is_pre_election: true
I20260925 11:11:29.904273 26507 raft_consensus.cc:2454] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da58acd969f401c97c1086bd8fdc4b6 in term 0.
I20260925 11:11:29.904574 26373 raft_consensus.cc:2454] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da58acd969f401c97c1086bd8fdc4b6 in term 0.
I20260925 11:11:29.904950 26175 leader_election.cc:302] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d3f5258368a44abb189fe58443fee5f, 5da58acd969f401c97c1086bd8fdc4b6; no voters:
I20260925 11:11:29.905454 26579 raft_consensus.cc:2781] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:29.905632 26579 raft_consensus.cc:482] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:29.905745 26579 raft_consensus.cc:3037] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:29.908079 26579 raft_consensus.cc:504] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.908702 26579 leader_election.cc:288] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [CANDIDATE]: Term 1 election: Requested vote from peers 3d3f5258368a44abb189fe58443fee5f (127.8.181.3:36513), 02ed89df45ac438c842c68147b3ba311 (127.8.181.2:45277)
I20260925 11:11:29.909268 26373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56595ad3f8534ad6b7c19885142a8f5c" candidate_uuid: "5da58acd969f401c97c1086bd8fdc4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ed89df45ac438c842c68147b3ba311"
I20260925 11:11:29.909256 26507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56595ad3f8534ad6b7c19885142a8f5c" candidate_uuid: "5da58acd969f401c97c1086bd8fdc4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d3f5258368a44abb189fe58443fee5f"
I20260925 11:11:29.909487 26373 raft_consensus.cc:3037] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:29.909536 26507 raft_consensus.cc:3037] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:29.911814 26373 raft_consensus.cc:2454] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da58acd969f401c97c1086bd8fdc4b6 in term 1.
I20260925 11:11:29.911847 26507 raft_consensus.cc:2454] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da58acd969f401c97c1086bd8fdc4b6 in term 1.
I20260925 11:11:29.912349 26174 leader_election.cc:302] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ed89df45ac438c842c68147b3ba311, 5da58acd969f401c97c1086bd8fdc4b6; no voters:
I20260925 11:11:29.912734 26579 raft_consensus.cc:2781] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:29.913291 26579 raft_consensus.cc:686] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 1 LEADER]: Becoming Leader. State: Replica: 5da58acd969f401c97c1086bd8fdc4b6, State: Running, Role: LEADER
I20260925 11:11:29.913823 26579 consensus_queue.cc:237] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } }
I20260925 11:11:29.919121 26095 catalog_manager.cc:5495] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da58acd969f401c97c1086bd8fdc4b6 (127.8.181.1). New cstate: current_term: 1 leader_uuid: "5da58acd969f401c97c1086bd8fdc4b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da58acd969f401c97c1086bd8fdc4b6" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 37893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 } health_report { overall_health: UNKNOWN } } }
I20260925 11:11:30.027248 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:30.021161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.8.181.1:37893)
0925 11:11:30.021754 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:30.021759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:30.021776 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:11:30.021867 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:30.021871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:30.021957 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:30.022100 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:30.022625 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.023397 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.024839 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.025291 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.025398 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:30.025873 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:30.026435 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:30.026664 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:30.026754 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 11:11:30.026829 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":338,"threads_started":1}
I20260925 11:11:30.035773 26373 raft_consensus.cc:1261] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 1 FOLLOWER]: Refusing update from remote peer 5da58acd969f401c97c1086bd8fdc4b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:30.035781 26507 raft_consensus.cc:1261] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 1 FOLLOWER]: Refusing update from remote peer 5da58acd969f401c97c1086bd8fdc4b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:30.036931 26582 consensus_queue.cc:1035] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d3f5258368a44abb189fe58443fee5f" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:30.037374 26579 consensus_queue.cc:1035] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ed89df45ac438c842c68147b3ba311" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 45277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:30.056624 26591 mvcc.cc:205] Tried to move back new op lower bound from 7333210890375286784 to 7333210889891147776. Current Snapshot: MvccSnapshot[applied={T|T < 7333210890375286784}]
I20260925 11:11:30.068713 26094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:30.069072 26094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:30.073539 26094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a02105a3283d45b1ad73c9a11e15c9a0: Sending DeleteTablet for 3 replicas of tablet 56595ad3f8534ad6b7c19885142a8f5c
I20260925 11:11:30.075074 26219 tablet_service.cc:1514] Processing DeleteTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:30 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260925 11:11:30.075071 26353 tablet_service.cc:1514] Processing DeleteTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:30 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:40208
I20260925 11:11:30.075347 26487 tablet_service.cc:1514] Processing DeleteTablet for tablet 56595ad3f8534ad6b7c19885142a8f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:30 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:49302
I20260925 11:11:30.077422 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26156
I20260925 11:11:30.091058 26601 tablet_replica.cc:330] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: stopping tablet replica
I20260925 11:11:30.091701 26601 raft_consensus.cc:2229] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:30.092367 26601 raft_consensus.cc:2258] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:30.094635 26600 tablet_replica.cc:330] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: stopping tablet replica
I20260925 11:11:30.095153 26600 raft_consensus.cc:2229] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:30.095737 26600 raft_consensus.cc:2258] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:30.097143 26601 ts_tablet_manager.cc:1905] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:30.097749 26600 ts_tablet_manager.cc:1905] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:30.101159 26599 tablet_replica.cc:330] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: stopping tablet replica
I20260925 11:11:30.101802 26599 raft_consensus.cc:2229] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:30.102602 26599 raft_consensus.cc:2258] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:30.104722 26599 ts_tablet_manager.cc:1905] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:30.104768 26601 ts_tablet_manager.cc:1918] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:30.104928 26601 log.cc:1196] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/56595ad3f8534ad6b7c19885142a8f5c
I20260925 11:11:30.105472 26601 ts_tablet_manager.cc:1939] T 56595ad3f8534ad6b7c19885142a8f5c P 3d3f5258368a44abb189fe58443fee5f: Deleting consensus metadata
I20260925 11:11:30.106163 26600 ts_tablet_manager.cc:1918] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:30.106355 26600 log.cc:1196] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/56595ad3f8534ad6b7c19885142a8f5c
I20260925 11:11:30.106990 26600 ts_tablet_manager.cc:1939] T 56595ad3f8534ad6b7c19885142a8f5c P 02ed89df45ac438c842c68147b3ba311: Deleting consensus metadata
I20260925 11:11:30.107483 26082 catalog_manager.cc:4841] TS 3d3f5258368a44abb189fe58443fee5f (127.8.181.3:36513): tablet 56595ad3f8534ad6b7c19885142a8f5c (table test-table [id=6663ed3028da4523bee8377050d42378]) successfully deleted
I20260925 11:11:30.109195 26081 catalog_manager.cc:4841] TS 02ed89df45ac438c842c68147b3ba311 (127.8.181.2:45277): tablet 56595ad3f8534ad6b7c19885142a8f5c (table test-table [id=6663ed3028da4523bee8377050d42378]) successfully deleted
I20260925 11:11:30.111995 26599 ts_tablet_manager.cc:1918] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:30.112205 26599 log.cc:1196] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/56595ad3f8534ad6b7c19885142a8f5c
I20260925 11:11:30.112818 26599 ts_tablet_manager.cc:1939] T 56595ad3f8534ad6b7c19885142a8f5c P 5da58acd969f401c97c1086bd8fdc4b6: Deleting consensus metadata
Sep 25 11:11:30 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:11:30.114753 26082 catalog_manager.cc:4841] TS 5da58acd969f401c97c1086bd8fdc4b6 (127.8.181.1:37893): tablet 56595ad3f8534ad6b7c19885142a8f5c (table test-table [id=6663ed3028da4523bee8377050d42378]) successfully deleted
I20260925 11:11:30.122570 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:30.078919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.8.181.1:37893)
0925 11:11:30.086013 (+ 7094us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:30.086022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:30.091402 (+ 5380us) server_negotiation.cc:407] Connection header received
0925 11:11:30.091734 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:30.091742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:30.091854 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:30.091957 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:30.094689 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.096261 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.098421 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.099120 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.099248 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:30.114390 (+ 15142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:30.114408 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:30.114434 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:30.114534 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:30.119033 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.119891 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.119903 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.119907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.120086 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.120549 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.120555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.120558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.121089 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:30.121444 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:30.121755 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 11:11:30.122076 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6896}
I20260925 11:11:30.123482 26279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:30.224813 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26156
I20260925 11:11:30.243642 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26290
Sep 25 11:11:30 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:11:30.261780 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:30.244490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.8.181.2:45277)
0925 11:11:30.244709 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:30.244717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:30.244740 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:11:30.245092 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:30.245097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:30.245182 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:30.245297 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:30.246664 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.247572 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.249097 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.249541 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.249686 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:30.255074 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:30.255093 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:30.255109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:30.255154 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:30.258222 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.259314 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.259323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.259326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.259480 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.259953 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.259966 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.259969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.260405 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:30.260767 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:30.261117 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 11:11:30.261407 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:11:30.262609 26413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:30.354287 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26290
I20260925 11:11:30.372288 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26424
Sep 25 11:11:30 dist-test-slave-6042 krb5kdc[26050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:11:30.390182 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:30.373153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.8.181.3:36513)
0925 11:11:30.373332 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:30.373338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:30.373422 (+ 84us) server_negotiation.cc:407] Connection header received
0925 11:11:30.373573 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:30.373578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:30.373653 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:30.373770 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:30.375306 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.376450 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.377852 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:30.378535 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:30.378660 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:30.383502 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:30.383517 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:30.383533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:30.383581 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:30.387393 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.388219 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.388226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.388229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.388373 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:30.388759 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:30.388762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:30.388764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:30.389132 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:30.389346 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:30.389700 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 11:11:30.389875 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:11:30.390987 26547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:30.482244 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26424
I20260925 11:11:30.499940 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26063
I20260925 11:11:30.501274 26125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:30.623103 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26063
2026-09-25T11:11:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T11:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:30Z Disabled control of system clock
I20260925 11:11:30.708145 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40153
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:31.053067 26610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:31.053373 26610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:31.053529 26610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:31.060889 26610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:31.061030 26610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:31.061115 26610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:31.061210 26610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:31.061299 26610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:31.070793 26610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:31.072324 26610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:31.074141 26610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:31.086601 26615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:31.086534 26618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:31.086606 26616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:31.087569 26610 server_base.cc:1034] running on GCE node
I20260925 11:11:31.088508 26610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:31.090101 26610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:31.091365 26610 hybrid_clock.cc:648] HybridClock initialized: now 1790334691091313 us; error 44 us; skew 500 ppm
I20260925 11:11:31.091789 26610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:31.094252 26610 webserver.cc:460] Webserver started at http://127.8.181.62:45999/ using document root <none> and password file <none>
I20260925 11:11:31.095077 26610 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:31.095213 26610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:31.095537 26610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:31.098652 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "bdb77fc0f9934f94a00784dd61a90284"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "d94df21b1b2970017b6e8bcc940d7ea5"
server_key_iv: "db3527fbfdf19db79d716c03b496a5f7"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.099511 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "bdb77fc0f9934f94a00784dd61a90284"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "d94df21b1b2970017b6e8bcc940d7ea5"
server_key_iv: "db3527fbfdf19db79d716c03b496a5f7"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.104832 26610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:31.108357 26624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:31.109717 26610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:31.109912 26610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "bdb77fc0f9934f94a00784dd61a90284"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "d94df21b1b2970017b6e8bcc940d7ea5"
server_key_iv: "db3527fbfdf19db79d716c03b496a5f7"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.110140 26610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:31.125981 26610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:31.126932 26610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:31.127235 26610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:31.146008 26610 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:40153
I20260925 11:11:31.146047 26675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:40153 every 8 connection(s)
I20260925 11:11:31.147446 26610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:31.151357 26676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:31.154044 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26610
I20260925 11:11:31.154238 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:31.154762 8916 external_mini_cluster.cc:1428] Setting key f367d83131035a2b5144a1e6be27548f
I20260925 11:11:31.165231 26676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: Bootstrap starting.
I20260925 11:11:31.169207 26676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:31.170637 26676 log.cc:824] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:31.171527 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:31.157036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34910 (local address 127.8.181.62:40153)
0925 11:11:31.158168 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:31.158182 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:31.158232 (+ 50us) server_negotiation.cc:407] Connection header received
0925 11:11:31.160627 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:31.160662 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:31.161334 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:31.161796 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:31.162384 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.164369 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.165920 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.167014 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.167214 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:31.168478 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:31.168488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:31.168508 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:31.169246 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:31.169365 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:31.169406 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:31.169516 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:31.169771 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 11:11:31.170115 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":349,"threads_started":1}
I20260925 11:11:31.173659 26676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: No bootstrap required, opened a new log
I20260925 11:11:31.181924 26676 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } }
I20260925 11:11:31.182309 26676 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:31.182433 26676 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb77fc0f9934f94a00784dd61a90284, State: Initialized, Role: FOLLOWER
I20260925 11:11:31.183081 26676 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } }
I20260925 11:11:31.183382 26676 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:31.183530 26676 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:31.183714 26676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:31.186018 26676 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } }
I20260925 11:11:31.186589 26676 leader_election.cc:302] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdb77fc0f9934f94a00784dd61a90284; no voters:
I20260925 11:11:31.187217 26676 leader_election.cc:288] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:31.187377 26681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:31.188154 26681 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [term 1 LEADER]: Becoming Leader. State: Replica: bdb77fc0f9934f94a00784dd61a90284, State: Running, Role: LEADER
I20260925 11:11:31.188657 26681 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } }
I20260925 11:11:31.189381 26676 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:31.192252 26683 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdb77fc0f9934f94a00784dd61a90284. Latest consensus state: current_term: 1 leader_uuid: "bdb77fc0f9934f94a00784dd61a90284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } } }
I20260925 11:11:31.193557 26683 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:31.193171 26682 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdb77fc0f9934f94a00784dd61a90284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdb77fc0f9934f94a00784dd61a90284" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40153 } } }
I20260925 11:11:31.193876 26682 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:31.199523 26691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:31.205107 26691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:31.214699 26691 catalog_manager.cc:1347] Generated new cluster ID: 68ae1869142741c58bd6f2bf038fd880
I20260925 11:11:31.214839 26691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:31.236326 26691 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:31.237789 26691 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:31.247380 26691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: Generated new TSK 0
I20260925 11:11:31.248263 26691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:31.252283 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--builtin_ntp_servers=127.8.181.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:31.603266 26700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:31.603602 26700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:31.603773 26700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:31.612310 26700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:31.612445 26700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:31.612633 26700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:31.623553 26700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:31.625007 26700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:31.626976 26700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:31.641806 26706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:31.641767 26708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:31.641860 26705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:31.648326 26700 server_base.cc:1034] running on GCE node
I20260925 11:11:31.649214 26700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:31.650379 26700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:31.651615 26700 hybrid_clock.cc:648] HybridClock initialized: now 1790334691651565 us; error 50 us; skew 500 ppm
I20260925 11:11:31.651975 26700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:31.654692 26700 webserver.cc:460] Webserver started at http://127.8.181.1:45653/ using document root <none> and password file <none>
I20260925 11:11:31.655440 26700 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:31.655557 26700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:31.655906 26700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:31.659049 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "5aba838075b6451d8130fdca8d40fb04"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "c4ef18a66732f629ac94c3ae55e7f05d"
server_key_iv: "f751a6f955b32ad8ae0d429b79105310"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.659857 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "5aba838075b6451d8130fdca8d40fb04"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "c4ef18a66732f629ac94c3ae55e7f05d"
server_key_iv: "f751a6f955b32ad8ae0d429b79105310"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.665194 26700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 11:11:31.669018 26714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:31.670761 26700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 11:11:31.670965 26700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "5aba838075b6451d8130fdca8d40fb04"
format_stamp: "Formatted at 2026-09-25 11:11:31 on dist-test-slave-6042"
server_key: "c4ef18a66732f629ac94c3ae55e7f05d"
server_key_iv: "f751a6f955b32ad8ae0d429b79105310"
server_key_version: "encryptionkey@0"
I20260925 11:11:31.671242 26700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:31.684693 26700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:31.685573 26700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:31.685870 26700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:31.686990 26700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:31.688619 26700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:31.688707 26700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:31.688855 26700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:31.688952 26700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:31.729034 26700 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:34277
I20260925 11:11:31.729214 26826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:34277 every 8 connection(s)
I20260925 11:11:31.730438 26700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:31.739513 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26700
I20260925 11:11:31.739711 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:31.740186 8916 external_mini_cluster.cc:1428] Setting key eec5328c4d18dc0386bee9847fcdda77
I20260925 11:11:31.745772 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:31.733342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:40009 (local address 127.8.181.62:40153)
0925 11:11:31.733843 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:31.733847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:31.735069 (+ 1222us) server_negotiation.cc:407] Connection header received
0925 11:11:31.737470 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:31.737477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:31.737582 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:31.737720 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:31.739009 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.739963 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.742099 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.742665 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.742832 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:31.744754 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:31.744760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:31.744764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:31.744785 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:31.744856 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:31.744877 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:31.744992 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:31.745294 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 11:11:31.745428 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260925 11:11:31.747452 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:31.733812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40153 (local address 127.8.181.1:40009)
0925 11:11:31.734883 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:31.734929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:31.737136 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:31.738002 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:31.738012 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:31.738109 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:31.738813 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:31.738833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.740140 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.740145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:31.741934 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:31.741948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:31.742950 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:31.742976 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:31.743249 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:31.744343 (+ 1094us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:31.744346 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:31.744394 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:31.744528 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:31.745063 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:31.745074 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:31.745081 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 11:11:31.745252 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 11:11:31.745694 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":324,"threads_started":1}
I20260925 11:11:31.748162 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--builtin_ntp_servers=127.8.181.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:31.749192 26827 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40153
I20260925 11:11:31.749548 26827 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:31.750432 26827 heartbeater.cc:511] Master 127.8.181.62:40153 requested a full tablet report, sending...
I20260925 11:11:31.752786 26641 ts_manager.cc:194] Registered new tserver with Master: 5aba838075b6451d8130fdca8d40fb04 (127.8.181.1:34277)
I20260925 11:11:31.754637 26641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.1:40009
W20260925 11:11:32.090576 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:32.090925 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:32.091059 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:32.098986 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:32.099107 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:32.099280 26831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:32.109071 26831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:32.110559 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:32.112471 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:32.125751 26837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:32.125842 26836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:32.126757 26839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:32.128120 26831 server_base.cc:1034] running on GCE node
I20260925 11:11:32.128711 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:32.129746 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:32.131019 26831 hybrid_clock.cc:648] HybridClock initialized: now 1790334692130945 us; error 57 us; skew 500 ppm
I20260925 11:11:32.131424 26831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:32.134112 26831 webserver.cc:460] Webserver started at http://127.8.181.2:32791/ using document root <none> and password file <none>
I20260925 11:11:32.135021 26831 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:32.135167 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:32.135537 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:32.138918 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "c6b433e2ec1044f7a70d1aa04fb669f3"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "2785360b09b35d4fa507fbd7150a3dd9"
server_key_iv: "5319e75e68a4948a5e856d2d382a993d"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.139820 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b433e2ec1044f7a70d1aa04fb669f3"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "2785360b09b35d4fa507fbd7150a3dd9"
server_key_iv: "5319e75e68a4948a5e856d2d382a993d"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.145816 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 11:11:32.150301 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.152022 26831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:32.152225 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "c6b433e2ec1044f7a70d1aa04fb669f3"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "2785360b09b35d4fa507fbd7150a3dd9"
server_key_iv: "5319e75e68a4948a5e856d2d382a993d"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.152479 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:32.171029 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:32.171919 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:32.172236 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:32.173252 26831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:32.174876 26831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:32.174999 26831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.175123 26831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:32.175222 26831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.209589 26831 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:37855
I20260925 11:11:32.209632 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:37855 every 8 connection(s)
I20260925 11:11:32.211001 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:32.216490 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26831
I20260925 11:11:32.216679 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:32.217185 8916 external_mini_cluster.cc:1428] Setting key 0daf1c21239977658f2dd1fd3f2017f3
I20260925 11:11:32.220546 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--builtin_ntp_servers=127.8.181.20:39647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:32.225271 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.213803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:34911 (local address 127.8.181.62:40153)
0925 11:11:32.214028 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.214035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.215556 (+ 1521us) server_negotiation.cc:407] Connection header received
0925 11:11:32.217487 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.217494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.217593 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.217725 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:32.219159 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.220193 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.221971 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.222735 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.222878 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.224324 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:32.224333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:32.224341 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:32.224370 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:32.224415 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:32.224433 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:32.224526 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.224742 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.224860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:11:32.226284 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.214318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40153 (local address 127.8.181.2:34911)
0925 11:11:32.215382 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.215423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.217159 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.217974 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.217986 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.218108 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:32.218904 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.218928 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.220352 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.220356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.221806 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.221819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.222961 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.222966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.223185 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.223990 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:32.223993 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:32.224035 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:32.224121 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:32.224532 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:32.224539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:32.224545 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.224694 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.225079 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":354,"threads_started":1}
I20260925 11:11:32.227844 26958 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40153
I20260925 11:11:32.228191 26958 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:32.229132 26958 heartbeater.cc:511] Master 127.8.181.62:40153 requested a full tablet report, sending...
I20260925 11:11:32.230794 26641 ts_manager.cc:194] Registered new tserver with Master: c6b433e2ec1044f7a70d1aa04fb669f3 (127.8.181.2:37855)
I20260925 11:11:32.231670 26641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.2:34911
W20260925 11:11:32.581351 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:32.581681 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:32.581849 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:32.590394 26961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:32.590572 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:32.590751 26961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:32.600531 26961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:39647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:40153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:32.601925 26961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:32.603757 26961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:32.618145 26967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:32.618062 26966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:32.618119 26969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:32.624742 26961 server_base.cc:1034] running on GCE node
I20260925 11:11:32.625649 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:32.626886 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:32.628202 26961 hybrid_clock.cc:648] HybridClock initialized: now 1790334692628144 us; error 48 us; skew 500 ppm
I20260925 11:11:32.628656 26961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:32.631292 26961 webserver.cc:460] Webserver started at http://127.8.181.3:41359/ using document root <none> and password file <none>
I20260925 11:11:32.632071 26961 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:32.632193 26961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:32.632603 26961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:32.635885 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "35767d03acf443318625bc21374d18c2"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "15191a2cf4a0e3e7084bbf87bb5052ab"
server_key_iv: "9d9019bc086defa222b75dba60d95289"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.636765 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "35767d03acf443318625bc21374d18c2"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "15191a2cf4a0e3e7084bbf87bb5052ab"
server_key_iv: "9d9019bc086defa222b75dba60d95289"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.642750 26961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:11:32.646842 26975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.648715 26961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:11:32.648936 26961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "35767d03acf443318625bc21374d18c2"
format_stamp: "Formatted at 2026-09-25 11:11:32 on dist-test-slave-6042"
server_key: "15191a2cf4a0e3e7084bbf87bb5052ab"
server_key_iv: "9d9019bc086defa222b75dba60d95289"
server_key_version: "encryptionkey@0"
I20260925 11:11:32.649211 26961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:32.664809 26961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:32.665760 26961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:32.666085 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:32.667258 26961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:32.668838 26961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:32.668968 26961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.669117 26961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:32.669204 26961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:32.704698 26961 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:46275
I20260925 11:11:32.704751 27087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:46275 every 8 connection(s)
I20260925 11:11:32.706307 26961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:32.709996 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 26961
I20260925 11:11:32.710191 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:32.710878 8916 external_mini_cluster.cc:1428] Setting key 3f333006de8ac9cd226195ad917a7881
I20260925 11:11:32.720578 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.709515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:36629 (local address 127.8.181.62:40153)
0925 11:11:32.709724 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.709728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.711571 (+ 1843us) server_negotiation.cc:407] Connection header received
0925 11:11:32.713638 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.713643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.713737 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.713832 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:32.715016 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.715761 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.717378 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.717823 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.717975 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.719731 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:32.719741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:32.719749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:32.719781 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:32.719828 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:32.719852 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:32.719918 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.720241 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.720325 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:11:32.721704 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.710196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:40153 (local address 127.8.181.3:36629)
0925 11:11:32.711324 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.711410 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.713375 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.714083 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.714091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.714183 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:32.714855 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.714874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.715947 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.715957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.717220 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.717232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.718080 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.718087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.718321 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.719239 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:32.719245 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:32.719309 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:32.719432 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:32.720019 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:32.720027 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:32.720033 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.720174 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.720532 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":371,"threads_started":1}
I20260925 11:11:32.723367 27088 heartbeater.cc:347] Connected to a master server at 127.8.181.62:40153
I20260925 11:11:32.723677 27088 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:32.724431 27088 heartbeater.cc:511] Master 127.8.181.62:40153 requested a full tablet report, sending...
I20260925 11:11:32.726112 26641 ts_manager.cc:194] Registered new tserver with Master: 35767d03acf443318625bc21374d18c2 (127.8.181.3:46275)
I20260925 11:11:32.727006 26641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.3:36629
I20260925 11:11:32.739617 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:32.750269 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.743845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.8.181.62:40153)
0925 11:11:32.744255 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.744265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.744786 (+ 521us) server_negotiation.cc:407] Connection header received
0925 11:11:32.745030 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.745034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.745113 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.745221 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:32.745690 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.746414 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.747912 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.748322 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.748462 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.749552 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:32.749557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:32.749560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:32.749580 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:32.749610 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:32.749623 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:32.749721 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.749870 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.749966 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:11:32.757830 26827 heartbeater.cc:503] Master 127.8.181.62:40153 was elected leader, sending a full tablet report...
I20260925 11:11:32.754881 26641 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:32.762713 26641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:32.796669 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.776191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.8.181.2:37855)
0925 11:11:32.787433 (+ 11242us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.787442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.787487 (+ 45us) server_negotiation.cc:407] Connection header received
0925 11:11:32.787614 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.787624 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.787900 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.788131 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:32.788843 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.790588 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.793812 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.795408 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.795543 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.795794 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.796114 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.796301 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11102,"thread_start_us":10991,"threads_started":1}
I20260925 11:11:32.796690 27101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.775973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:37855 (local address 127.0.0.1:39074)
0925 11:11:32.786403 (+ 10430us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.786490 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.786806 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.788145 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.788153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.788184 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:32.788613 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.788628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.790746 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.790751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.793572 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.793581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.795615 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.795623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.795844 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.795897 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.796058 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.796279 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10002,"mutex_wait_us":65,"thread_start_us":11429,"threads_started":1}
I20260925 11:11:32.796797 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.776107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59704)
0925 11:11:32.787638 (+ 11531us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.787655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.787834 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.788705 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.788712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.788744 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:32.789173 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.789184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.791137 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.791145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.793551 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.793567 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.795949 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.795953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.796145 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.796176 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.796268 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.796387 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11425,"mutex_wait_us":61,"thread_start_us":9706,"threads_started":1}
I20260925 11:11:32.797937 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.776802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.8.181.1:34277)
0925 11:11:32.787366 (+ 10564us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.787378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.787786 (+ 408us) server_negotiation.cc:407] Connection header received
0925 11:11:32.788017 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.788022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.788254 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.788445 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:32.789403 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.790974 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.793810 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.795398 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.795543 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.795826 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.797315 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.797631 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10357,"thread_start_us":10210,"threads_started":1}
I20260925 11:11:32.801236 26893 tablet_service.cc:1467] Processing CreateTablet for tablet c46a53307d0c4ae9890dad7b75097072 (DEFAULT_TABLE table=test-table [id=76f784f31d4f4c8d903b5e3ca23c52e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:32.801605 26762 tablet_service.cc:1467] Processing CreateTablet for tablet c46a53307d0c4ae9890dad7b75097072 (DEFAULT_TABLE table=test-table [id=76f784f31d4f4c8d903b5e3ca23c52e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:32.802762 26893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46a53307d0c4ae9890dad7b75097072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:32.803076 26762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46a53307d0c4ae9890dad7b75097072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:32.805258 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.788212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.8.181.3:46275)
0925 11:11:32.795046 (+ 6834us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.795056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.795091 (+ 35us) server_negotiation.cc:407] Connection header received
0925 11:11:32.795179 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.795184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.795370 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.795584 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:32.797478 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.798833 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.802126 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.803595 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.803747 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.804641 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.804716 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.804877 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6644,"thread_start_us":6484,"threads_started":1}
I20260925 11:11:32.805581 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.788476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46275 (local address 127.0.0.1:51440)
0925 11:11:32.792211 (+ 3735us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.792240 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.792390 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.796888 (+ 4498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.796893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.796919 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:32.797279 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.797292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.799033 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.799038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.801857 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.801877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.803821 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.803829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.804016 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.804053 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.804145 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.804294 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3550,"thread_start_us":4013,"threads_started":1}
I20260925 11:11:32.811015 27023 tablet_service.cc:1467] Processing CreateTablet for tablet c46a53307d0c4ae9890dad7b75097072 (DEFAULT_TABLE table=test-table [id=76f784f31d4f4c8d903b5e3ca23c52e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:32.812763 27023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46a53307d0c4ae9890dad7b75097072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:32.819278 27106 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Bootstrap starting.
I20260925 11:11:32.820147 27107 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Bootstrap starting.
I20260925 11:11:32.823518 27106 tablet_bootstrap.cc:654] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:32.824821 27106 log.cc:824] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:32.826577 27107 tablet_bootstrap.cc:654] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:32.827677 27106 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: No bootstrap required, opened a new log
I20260925 11:11:32.827829 27107 log.cc:824] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:32.828014 27106 ts_tablet_manager.cc:1397] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 11:11:32.828717 27109 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Bootstrap starting.
I20260925 11:11:32.831708 27107 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: No bootstrap required, opened a new log
I20260925 11:11:32.831997 27107 ts_tablet_manager.cc:1397] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260925 11:11:32.833227 27109 tablet_bootstrap.cc:654] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:32.834622 27109 log.cc:824] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:32.838146 27109 tablet_bootstrap.cc:492] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: No bootstrap required, opened a new log
I20260925 11:11:32.838454 27109 ts_tablet_manager.cc:1397] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260925 11:11:32.839344 27106 raft_consensus.cc:348] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.839854 27106 raft_consensus.cc:374] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:32.839965 27106 raft_consensus.cc:729] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aba838075b6451d8130fdca8d40fb04, State: Initialized, Role: FOLLOWER
I20260925 11:11:32.840500 27107 raft_consensus.cc:348] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.840698 27106 consensus_queue.cc:260] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.840932 27107 raft_consensus.cc:374] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:32.841037 27107 raft_consensus.cc:729] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b433e2ec1044f7a70d1aa04fb669f3, State: Initialized, Role: FOLLOWER
I20260925 11:11:32.841609 27107 consensus_queue.cc:260] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.842962 26958 heartbeater.cc:503] Master 127.8.181.62:40153 was elected leader, sending a full tablet report...
I20260925 11:11:32.843679 27107 ts_tablet_manager.cc:1428] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260925 11:11:32.846139 27106 ts_tablet_manager.cc:1428] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260925 11:11:32.852272 27109 raft_consensus.cc:348] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.852828 27109 raft_consensus.cc:374] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:32.852959 27109 raft_consensus.cc:729] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35767d03acf443318625bc21374d18c2, State: Initialized, Role: FOLLOWER
I20260925 11:11:32.853628 27109 consensus_queue.cc:260] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.854934 27088 heartbeater.cc:503] Master 127.8.181.62:40153 was elected leader, sending a full tablet report...
I20260925 11:11:32.855429 27109 ts_tablet_manager.cc:1428] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
I20260925 11:11:32.946800 27113 raft_consensus.cc:482] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:32.947279 27113 raft_consensus.cc:504] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.949268 27113 leader_election.cc:288] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aba838075b6451d8130fdca8d40fb04 (127.8.181.1:34277), 35767d03acf443318625bc21374d18c2 (127.8.181.3:46275)
W20260925 11:11:32.958601 27089 tablet.cc:2369] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:32.958683 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.950145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:38101 (local address 127.8.181.1:34277)
0925 11:11:32.950389 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.950397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.951305 (+ 908us) server_negotiation.cc:407] Connection header received
0925 11:11:32.951506 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.951515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.951614 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.951709 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:32.952372 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.953295 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.956525 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.957730 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.957874 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.957991 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.958259 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.958399 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 11:11:32.958815 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.949706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.8.181.2:38101)
0925 11:11:32.951158 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.951183 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.951356 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.951745 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.951750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.951776 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:32.952206 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.952220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.953439 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.953446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.956286 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.956296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.957951 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.957957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.958091 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.958118 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.958199 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.958362 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":175,"thread_start_us":1300,"threads_started":1}
I20260925 11:11:32.959661 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.949804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:35159 (local address 127.8.181.3:46275)
0925 11:11:32.950014 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.950022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.951569 (+ 1547us) server_negotiation.cc:407] Connection header received
0925 11:11:32.951749 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.951754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.951829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.951927 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:32.952594 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.953464 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.956980 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.958390 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.958566 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.958660 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:32.959249 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 11:11:32.959382 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:11:32.959710 27115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.949742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46275 (local address 127.8.181.2:35159)
0925 11:11:32.951439 (+ 1697us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:32.951459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:32.951594 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:32.951954 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:32.951957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:32.951980 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:32.952403 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.952415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.953593 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.953598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.956263 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:32.956288 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.958812 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.958818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:32.959009 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.959044 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 11:11:32.959184 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 11:11:32.959321 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"mutex_wait_us":141,"thread_start_us":840,"threads_started":1}
I20260925 11:11:32.960115 26782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46a53307d0c4ae9890dad7b75097072" candidate_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aba838075b6451d8130fdca8d40fb04" is_pre_election: true
I20260925 11:11:32.960628 26782 raft_consensus.cc:2454] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b433e2ec1044f7a70d1aa04fb669f3 in term 0.
I20260925 11:11:32.960686 27043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46a53307d0c4ae9890dad7b75097072" candidate_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35767d03acf443318625bc21374d18c2" is_pre_election: true
I20260925 11:11:32.961228 27043 raft_consensus.cc:2454] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b433e2ec1044f7a70d1aa04fb669f3 in term 0.
I20260925 11:11:32.961328 26846 leader_election.cc:302] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aba838075b6451d8130fdca8d40fb04, c6b433e2ec1044f7a70d1aa04fb669f3; no voters:
I20260925 11:11:32.961763 27113 raft_consensus.cc:2781] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:32.961894 27113 raft_consensus.cc:482] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:32.962077 27113 raft_consensus.cc:3037] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 0 FOLLOWER]: Advancing to term 1
W20260925 11:11:32.963157 26959 tablet.cc:2369] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:32.964570 27113 raft_consensus.cc:504] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.965268 27113 leader_election.cc:288] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5aba838075b6451d8130fdca8d40fb04 (127.8.181.1:34277), 35767d03acf443318625bc21374d18c2 (127.8.181.3:46275)
I20260925 11:11:32.965808 26782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46a53307d0c4ae9890dad7b75097072" candidate_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aba838075b6451d8130fdca8d40fb04"
I20260925 11:11:32.965852 27043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46a53307d0c4ae9890dad7b75097072" candidate_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35767d03acf443318625bc21374d18c2"
I20260925 11:11:32.966113 27043 raft_consensus.cc:3037] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:32.966101 26782 raft_consensus.cc:3037] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:32.968488 27043 raft_consensus.cc:2454] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b433e2ec1044f7a70d1aa04fb669f3 in term 1.
I20260925 11:11:32.968549 26782 raft_consensus.cc:2454] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b433e2ec1044f7a70d1aa04fb669f3 in term 1.
I20260925 11:11:32.969059 26846 leader_election.cc:302] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35767d03acf443318625bc21374d18c2, 5aba838075b6451d8130fdca8d40fb04, c6b433e2ec1044f7a70d1aa04fb669f3; no voters:
I20260925 11:11:32.969415 27113 raft_consensus.cc:2781] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:32.969903 27113 raft_consensus.cc:686] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 1 LEADER]: Becoming Leader. State: Replica: c6b433e2ec1044f7a70d1aa04fb669f3, State: Running, Role: LEADER
I20260925 11:11:32.970532 27113 consensus_queue.cc:237] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } }
I20260925 11:11:32.975606 26641 catalog_manager.cc:5495] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b433e2ec1044f7a70d1aa04fb669f3 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b433e2ec1044f7a70d1aa04fb669f3" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 } health_report { overall_health: UNKNOWN } } }
W20260925 11:11:32.983098 26828 tablet.cc:2369] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:33.001055 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:32.995537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39086 (local address 127.8.181.2:37855)
0925 11:11:32.995719 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:32.995724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:32.995862 (+ 138us) server_negotiation.cc:407] Connection header received
0925 11:11:32.996140 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:32.996145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:32.996248 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:32.996381 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:32.996715 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.997390 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.998960 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:32.999389 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:32.999490 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:32.999967 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:33.000524 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:33.000635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:33.000773 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 11:11:33.000837 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 11:11:33.010063 26782 raft_consensus.cc:1261] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 1 FOLLOWER]: Refusing update from remote peer c6b433e2ec1044f7a70d1aa04fb669f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:33.010092 27043 raft_consensus.cc:1261] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 1 FOLLOWER]: Refusing update from remote peer c6b433e2ec1044f7a70d1aa04fb669f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:33.011217 27113 consensus_queue.cc:1035] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aba838075b6451d8130fdca8d40fb04" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:33.011536 27117 consensus_queue.cc:1035] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35767d03acf443318625bc21374d18c2" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:33.022076 27120 mvcc.cc:205] Tried to move back new op lower bound from 7333210902557704192 to 7333210902411485184. Current Snapshot: MvccSnapshot[applied={T|T < 7333210902557704192}]
I20260925 11:11:33.025664 27121 mvcc.cc:205] Tried to move back new op lower bound from 7333210902557704192 to 7333210902411485184. Current Snapshot: MvccSnapshot[applied={T|T < 7333210902557704192}]
I20260925 11:11:33.027382 27122 mvcc.cc:205] Tried to move back new op lower bound from 7333210902557704192 to 7333210902411485184. Current Snapshot: MvccSnapshot[applied={T|T < 7333210902557704192}]
I20260925 11:11:33.039300 26641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:33.039548 26641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:33.043253 26641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdb77fc0f9934f94a00784dd61a90284: Sending DeleteTablet for 3 replicas of tablet c46a53307d0c4ae9890dad7b75097072
I20260925 11:11:33.044593 26893 tablet_service.cc:1514] Processing DeleteTablet for tablet c46a53307d0c4ae9890dad7b75097072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:33 UTC) from {username='slave'} at 127.0.0.1:39074
I20260925 11:11:33.044760 26762 tablet_service.cc:1514] Processing DeleteTablet for tablet c46a53307d0c4ae9890dad7b75097072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:33 UTC) from {username='slave'} at 127.0.0.1:59704
I20260925 11:11:33.045524 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26700
I20260925 11:11:33.045491 27023 tablet_service.cc:1514] Processing DeleteTablet for tablet c46a53307d0c4ae9890dad7b75097072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:33 UTC) from {username='slave'} at 127.0.0.1:51440
I20260925 11:11:33.057126 27130 tablet_replica.cc:330] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: stopping tablet replica
I20260925 11:11:33.058482 27131 tablet_replica.cc:330] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: stopping tablet replica
I20260925 11:11:33.057840 27130 raft_consensus.cc:2229] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:33.059034 27131 raft_consensus.cc:2229] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:33.059134 27130 raft_consensus.cc:2258] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:33.060384 27131 raft_consensus.cc:2258] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:33.061283 27130 ts_tablet_manager.cc:1905] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:33.062309 27131 ts_tablet_manager.cc:1905] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:33.066831 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.046307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.8.181.1:34277)
0925 11:11:33.046564 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:33.046573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:33.056048 (+ 9475us) server_negotiation.cc:407] Connection header received
0925 11:11:33.056285 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:33.056294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:33.056485 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:33.056652 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:33.057756 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.059680 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.061952 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.062600 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.062706 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:33.064855 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:33.064868 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:33.064885 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:33.065651 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:33.065712 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:33.065737 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:33.065821 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:33.066257 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 11:11:33.066380 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:11:33.068131 26822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:33.071136 27132 tablet_replica.cc:330] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: stopping tablet replica
I20260925 11:11:33.071632 27132 raft_consensus.cc:2229] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:33.072211 27132 raft_consensus.cc:2258] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:33.073623 27131 ts_tablet_manager.cc:1918] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:33.073829 27131 log.cc:1196] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/c46a53307d0c4ae9890dad7b75097072
I20260925 11:11:33.074402 27131 ts_tablet_manager.cc:1939] T c46a53307d0c4ae9890dad7b75097072 P c6b433e2ec1044f7a70d1aa04fb669f3: Deleting consensus metadata
I20260925 11:11:33.074620 27132 ts_tablet_manager.cc:1905] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:33.077013 26627 catalog_manager.cc:4841] TS c6b433e2ec1044f7a70d1aa04fb669f3 (127.8.181.2:37855): tablet c46a53307d0c4ae9890dad7b75097072 (table test-table [id=76f784f31d4f4c8d903b5e3ca23c52e9]) successfully deleted
I20260925 11:11:33.082526 27132 ts_tablet_manager.cc:1918] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:33.082717 27132 log.cc:1196] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/c46a53307d0c4ae9890dad7b75097072
I20260925 11:11:33.083266 27132 ts_tablet_manager.cc:1939] T c46a53307d0c4ae9890dad7b75097072 P 35767d03acf443318625bc21374d18c2: Deleting consensus metadata
I20260925 11:11:33.085323 26627 catalog_manager.cc:4841] TS 35767d03acf443318625bc21374d18c2 (127.8.181.3:46275): tablet c46a53307d0c4ae9890dad7b75097072 (table test-table [id=76f784f31d4f4c8d903b5e3ca23c52e9]) successfully deleted
I20260925 11:11:33.170432 27130 ts_tablet_manager.cc:1918] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:33.170723 27130 log.cc:1196] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/c46a53307d0c4ae9890dad7b75097072
I20260925 11:11:33.171226 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26700
I20260925 11:11:33.171391 27130 ts_tablet_manager.cc:1939] T c46a53307d0c4ae9890dad7b75097072 P 5aba838075b6451d8130fdca8d40fb04: Deleting consensus metadata
W20260925 11:11:33.190586 26625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.181.1:34277 (error 108)
I20260925 11:11:33.190729 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26831
I20260925 11:11:33.195508 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.194996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59728)
0925 11:11:33.195186 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.195319 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 11:11:33.195799 26625 catalog_manager.cc:4584] TS 5aba838075b6451d8130fdca8d40fb04 (127.8.181.1:34277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c46a53307d0c4ae9890dad7b75097072: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
I20260925 11:11:33.200069 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.192455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.8.181.2:37855)
0925 11:11:33.192694 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:33.192702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:33.192726 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:11:33.192812 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:33.192820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:33.192960 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:33.193094 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:33.194008 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.195186 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.196697 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.197170 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.197349 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:33.198532 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:33.198544 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:33.198562 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:33.199204 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:33.199251 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:33.199276 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:33.199395 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:33.199553 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 11:11:33.199661 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 11:11:33.200749 26953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:33.249683 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.249144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59742)
0925 11:11:33.249394 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.249515 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 11:11:33.250982 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.250522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59748)
0925 11:11:33.250716 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.250815 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 11:11:33.311529 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.311037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59754)
0925 11:11:33.311273 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.311374 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 11:11:33.312604 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.312241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59758)
0925 11:11:33.312396 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.312470 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 11:11:33.316697 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26831
I20260925 11:11:33.335745 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26961
I20260925 11:11:33.343137 27104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.336564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.8.181.3:46275)
0925 11:11:33.336787 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:33.336795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:33.336865 (+ 70us) server_negotiation.cc:407] Connection header received
0925 11:11:33.337156 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:33.337162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:33.337257 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:33.337372 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:33.337803 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.338641 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.339927 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:33.340316 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:33.340457 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:33.341586 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:33.341595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:33.341609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:33.342165 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:33.342212 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:33.342230 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:33.342349 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:33.342714 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 11:11:33.342822 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 11:11:33.343986 27083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:33.393121 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.392623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59770)
0925 11:11:33.392841 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.392928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 11:11:33.394286 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:33.393896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:34277 (local address 127.0.0.1:59786)
0925 11:11:33.394093 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:33.394171 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 11:11:33.434556 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26961
I20260925 11:11:33.451880 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 26610
I20260925 11:11:33.453227 26671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:33.563921 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 26610
2026-09-25T11:11:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2937 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:33 dist-test-slave-6042 krb5kdc[27141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:33 dist-test-slave-6042 krb5kdc[27141](info): set up 2 sockets
Sep 25 11:11:33 dist-test-slave-6042 krb5kdc[27141](info): commencing operation
krb5kdc: starting...
W20260925 11:11:34.715946 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:34 dist-test-slave-6042 krb5kdc[27141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:11:34.989749 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35285
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:35.329155 27154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:35.329455 27154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:35.329610 27154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:35.337714 27154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:35.337827 27154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:35.337935 27154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:35.338007 27154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:35.338085 27154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:35.348338 27154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:35285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:35.349886 27154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:35.351518 27154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:35.364169 27159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:35.364142 27162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:35.364079 27160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:35.364711 27154 server_base.cc:1034] running on GCE node
I20260925 11:11:35.365547 27154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:35.367120 27154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:35.368394 27154 hybrid_clock.cc:648] HybridClock initialized: now 1790334695368343 us; error 60 us; skew 500 ppm
I20260925 11:11:35.368815 27154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:35 dist-test-slave-6042 krb5kdc[27141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:35.379792 27154 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:11:35.381547 27154 webserver.cc:460] Webserver started at http://127.8.181.62:35657/ using document root <none> and password file <none>
I20260925 11:11:35.382258 27154 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:35.382364 27154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:35.382720 27154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:35.385747 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "844ff74037944fcd812bb7c408dabb37"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "340405c1494a5938bc865330bd5cc4a8"
server_key_iv: "c390e75abbde32c770ff1300df7f47d5"
server_key_version: "encryptionkey@0"
I20260925 11:11:35.386605 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "844ff74037944fcd812bb7c408dabb37"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "340405c1494a5938bc865330bd5cc4a8"
server_key_iv: "c390e75abbde32c770ff1300df7f47d5"
server_key_version: "encryptionkey@0"
I20260925 11:11:35.391870 27154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:11:35.395360 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:35.396821 27154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 11:11:35.397032 27154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "844ff74037944fcd812bb7c408dabb37"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "340405c1494a5938bc865330bd5cc4a8"
server_key_iv: "c390e75abbde32c770ff1300df7f47d5"
server_key_version: "encryptionkey@0"
I20260925 11:11:35.397279 27154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:35.424485 27154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:35.429185 27154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:35.429512 27154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:35.449394 27154 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:35285
I20260925 11:11:35.449429 27220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:35285 every 8 connection(s)
I20260925 11:11:35.450996 27154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:35.455005 27221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:35.456995 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27154
I20260925 11:11:35.457221 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:35.457757 8916 external_mini_cluster.cc:1428] Setting key 1e2e2feb6360731296ac791a9776ee82
I20260925 11:11:35.466823 27221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: Bootstrap starting.
I20260925 11:11:35.470088 27221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:35.471199 27221 log.cc:824] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:11:35 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:35.474144 27221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: No bootstrap required, opened a new log
I20260925 11:11:35.481475 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:35.459885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.8.181.62:35285)
0925 11:11:35.460755 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:35.460767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:35.460811 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:11:35.462631 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:35.462657 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:35.463057 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:35.463580 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:35.465332 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:35.466769 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:35.468146 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:35.468834 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:35.469033 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:35.473701 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:35.473710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:35.473724 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:35.473759 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:35.477186 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:35.477980 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:35.477987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:35.477993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:35.478155 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:35.478580 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:35.478584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:35.478596 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:35.479171 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:35.479482 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:35.479979 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 11:11:35.480412 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":403,"threads_started":1}
I20260925 11:11:35.482640 27221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } }
I20260925 11:11:35.483042 27221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:35.483178 27221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844ff74037944fcd812bb7c408dabb37, State: Initialized, Role: FOLLOWER
I20260925 11:11:35.483764 27221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } }
I20260925 11:11:35.484014 27221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:35.484154 27221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:35.484334 27221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:35.486671 27221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } }
I20260925 11:11:35.487185 27221 leader_election.cc:302] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 844ff74037944fcd812bb7c408dabb37; no voters:
I20260925 11:11:35.487807 27221 leader_election.cc:288] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:35.488000 27226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:35.488754 27226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [term 1 LEADER]: Becoming Leader. State: Replica: 844ff74037944fcd812bb7c408dabb37, State: Running, Role: LEADER
I20260925 11:11:35.489378 27226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } }
I20260925 11:11:35.490077 27221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:35.492233 27228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 844ff74037944fcd812bb7c408dabb37. Latest consensus state: current_term: 1 leader_uuid: "844ff74037944fcd812bb7c408dabb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } } }
I20260925 11:11:35.492653 27228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:35.493311 27227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "844ff74037944fcd812bb7c408dabb37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "844ff74037944fcd812bb7c408dabb37" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35285 } } }
I20260925 11:11:35.493646 27227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:35.496282 27233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:35.501394 27233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:35.510399 27233 catalog_manager.cc:1347] Generated new cluster ID: bf5bc10f603541b6b2ec7b344e1337de
I20260925 11:11:35.510564 27233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:35.528151 27233 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:35.529573 27233 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:35.537081 27233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: Generated new TSK 0
I20260925 11:11:35.538010 27233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:11:35.599648 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:35.948457 27248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:35.948758 27248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:35.948854 27248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:35.956377 27248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:35.956496 27248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:35.956609 27248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:35.967406 27248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:35.968715 27248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:35.969967 27248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:35.983628 27256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:35.983688 27253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:35.983906 27254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:35.984573 27248 server_base.cc:1034] running on GCE node
I20260925 11:11:35.985282 27248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:35.986346 27248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:35.987603 27248 hybrid_clock.cc:648] HybridClock initialized: now 1790334695987566 us; error 75 us; skew 500 ppm
I20260925 11:11:35.987988 27248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:35 dist-test-slave-6042 krb5kdc[27141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:35.991551 27248 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:11:35.993285 27248 webserver.cc:460] Webserver started at http://127.8.181.1:35881/ using document root <none> and password file <none>
I20260925 11:11:35.994083 27248 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:35.994246 27248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:35.994665 27248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:35.998042 27248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "b5ecb268bf3441248a2176e1b5a15a2c"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "0abfae8785896d92fabf592bf7daf19f"
server_key_iv: "849dd015f27b5fa7ef7db84c17adcc49"
server_key_version: "encryptionkey@0"
I20260925 11:11:35.999023 27248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "b5ecb268bf3441248a2176e1b5a15a2c"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "0abfae8785896d92fabf592bf7daf19f"
server_key_iv: "849dd015f27b5fa7ef7db84c17adcc49"
server_key_version: "encryptionkey@0"
I20260925 11:11:36.005052 27248 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:11:36.009349 27263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.011060 27248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 11:11:36.011260 27248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "b5ecb268bf3441248a2176e1b5a15a2c"
format_stamp: "Formatted at 2026-09-25 11:11:35 on dist-test-slave-6042"
server_key: "0abfae8785896d92fabf592bf7daf19f"
server_key_iv: "849dd015f27b5fa7ef7db84c17adcc49"
server_key_version: "encryptionkey@0"
I20260925 11:11:36.011497 27248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:36.025164 27248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:36.029847 27248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:36.030186 27248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:36.031276 27248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:36.032691 27248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:36.032799 27248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.032963 27248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:36.033043 27248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.068904 27248 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:36481
I20260925 11:11:36.068953 27375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:36481 every 8 connection(s)
I20260925 11:11:36.070236 27248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:36.077041 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27248
I20260925 11:11:36.077250 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:36.077682 8916 external_mini_cluster.cc:1428] Setting key 209584adafa347b8d0957301ddf0dbb5
Sep 25 11:11:36 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:36.091109 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:36.073039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:44607 (local address 127.8.181.62:35285)
0925 11:11:36.073469 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:36.073474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:36.074359 (+ 885us) server_negotiation.cc:407] Connection header received
0925 11:11:36.076231 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:36.076235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:36.076303 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:36.076399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:36.078175 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.079002 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.080610 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.081034 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.081176 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:36.084882 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:36.084887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:36.084891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:36.084927 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:36.087601 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:36.088445 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:36.088450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:36.088452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:36.088583 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:36.089058 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:36.089068 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:36.089070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:36.089348 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:36.089532 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:36.090371 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 11:11:36.090655 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":216,"threads_started":1}
I20260925 11:11:36.092133 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:36.073498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35285 (local address 127.8.181.1:44607)
0925 11:11:36.074194 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:36.074232 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:36.075916 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:36.076620 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:36.076629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:36.077286 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:36.077949 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:36.078003 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.079174 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.079178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:36.080444 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:36.080456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.081290 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.081297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:36.081548 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:36.082284 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:36.082307 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:36.084612 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:36.087797 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:36.087807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:36.087832 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:36.088290 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:36.088704 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:36.088709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:36.088713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:36.088921 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:36.089485 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:36.089495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:36.089940 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 11:11:36.090234 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 11:11:36.090868 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260925 11:11:36.093861 27376 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35285
I20260925 11:11:36.094231 27376 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:36.095027 27376 heartbeater.cc:511] Master 127.8.181.62:35285 requested a full tablet report, sending...
I20260925 11:11:36.097314 27186 ts_manager.cc:194] Registered new tserver with Master: b5ecb268bf3441248a2176e1b5a15a2c (127.8.181.1:36481)
I20260925 11:11:36.099546 27186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:44607
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:11:36.138573 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:36.472009 27382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:36.472309 27382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:36.472402 27382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:36.479775 27382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:36.479899 27382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:36.480017 27382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:36.489936 27382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:36.491477 27382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:36.492717 27382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:36.505798 27387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:36.505874 27388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:36.505841 27390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:36.506726 27382 server_base.cc:1034] running on GCE node
I20260925 11:11:36.507285 27382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:36.508194 27382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:36.509392 27382 hybrid_clock.cc:648] HybridClock initialized: now 1790334696509333 us; error 57 us; skew 500 ppm
I20260925 11:11:36.509668 27382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:36 dist-test-slave-6042 krb5kdc[27141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334696, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:36.512842 27382 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:11:36.514230 27382 webserver.cc:460] Webserver started at http://127.8.181.2:43355/ using document root <none> and password file <none>
I20260925 11:11:36.514950 27382 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:36.515128 27382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:36.515455 27382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:36.518416 27382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "2a32f9df41e64260b70d260711405c88"
format_stamp: "Formatted at 2026-09-25 11:11:36 on dist-test-slave-6042"
server_key: "7d7e1da80eb83033fb6d1f39e27fdae2"
server_key_iv: "a398c6b0019b9c3a32946e95992bbdee"
server_key_version: "encryptionkey@0"
I20260925 11:11:36.519269 27382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "2a32f9df41e64260b70d260711405c88"
format_stamp: "Formatted at 2026-09-25 11:11:36 on dist-test-slave-6042"
server_key: "7d7e1da80eb83033fb6d1f39e27fdae2"
server_key_iv: "a398c6b0019b9c3a32946e95992bbdee"
server_key_version: "encryptionkey@0"
I20260925 11:11:36.525621 27382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:11:36.531116 27397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.533618 27382 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260925 11:11:36.533828 27382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "2a32f9df41e64260b70d260711405c88"
format_stamp: "Formatted at 2026-09-25 11:11:36 on dist-test-slave-6042"
server_key: "7d7e1da80eb83033fb6d1f39e27fdae2"
server_key_iv: "a398c6b0019b9c3a32946e95992bbdee"
server_key_version: "encryptionkey@0"
I20260925 11:11:36.534070 27382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:36.548395 27382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:36.552940 27382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:36.553267 27382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:36.554253 27382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:36.555650 27382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:36.555752 27382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.555894 27382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:36.555963 27382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:36.588344 27382 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:44817
I20260925 11:11:36.588392 27509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:44817 every 8 connection(s)
I20260925 11:11:36.589660 27382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:36.594357 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27382
I20260925 11:11:36.594584 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:36.595144 8916 external_mini_cluster.cc:1428] Setting key 5754378224921a19d1473513c855f0c8
Sep 25 11:11:36 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334696, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:36.613379 27512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:36.592693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:53535 (local address 127.8.181.62:35285)
0925 11:11:36.593225 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:36.593230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:36.594146 (+ 916us) server_negotiation.cc:407] Connection header received
0925 11:11:36.596560 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:36.596565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:36.596632 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:36.596731 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:36.598798 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.599662 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.602256 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.602797 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.602977 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:36.607514 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:36.607519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:36.607523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:36.607562 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:36.610142 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:36.611009 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:36.611013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:36.611016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:36.611148 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:36.611614 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:36.611618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:36.611619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:36.611882 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:36.612035 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:36.612745 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 11:11:36.612972 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
I20260925 11:11:36.614434 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:36.593202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35285 (local address 127.8.181.2:53535)
0925 11:11:36.593958 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:36.594009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:36.596264 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:36.596998 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:36.597009 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:36.597707 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:36.598565 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:36.598593 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.599815 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.599823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:36.601994 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:36.602010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:36.603037 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:36.603046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:36.603382 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:36.604249 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:36.604284 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:36.607239 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:36.610297 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:36.610307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:36.610326 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:36.610871 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:36.611257 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:36.611263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:36.611267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:36.611460 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:36.612008 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:36.612017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:36.612389 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 11:11:36.612651 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 11:11:36.613260 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
I20260925 11:11:36.616197 27510 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35285
I20260925 11:11:36.616559 27510 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:36.617476 27510 heartbeater.cc:511] Master 127.8.181.62:35285 requested a full tablet report, sending...
I20260925 11:11:36.619379 27186 ts_manager.cc:194] Registered new tserver with Master: 2a32f9df41e64260b70d260711405c88 (127.8.181.2:44817)
I20260925 11:11:36.620733 27186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:53535
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:11:36.658390 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:36.997174 27516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:36.997484 27516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:36.997639 27516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:37.005378 27516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:37.005528 27516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:37.005724 27516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:37.015938 27516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:37.017360 27516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:37.018788 27516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:37.032935 27524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:37.032984 27522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:37.032984 27521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:37.034206 27516 server_base.cc:1034] running on GCE node
I20260925 11:11:37.034897 27516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:37.035967 27516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:37.037191 27516 hybrid_clock.cc:648] HybridClock initialized: now 1790334697037147 us; error 43 us; skew 500 ppm
I20260925 11:11:37.037510 27516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:37 dist-test-slave-6042 krb5kdc[27141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334697, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:37.040833 27516 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:11:37.042459 27516 webserver.cc:460] Webserver started at http://127.8.181.3:41675/ using document root <none> and password file <none>
I20260925 11:11:37.043251 27516 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:37.043395 27516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:37.043705 27516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:37.046777 27516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "e16ccfcede2e47c5808a92231c9ee564"
format_stamp: "Formatted at 2026-09-25 11:11:37 on dist-test-slave-6042"
server_key: "190959602705eda3fdd7de282c88cdb4"
server_key_iv: "a7887604e41a15137c8960efb33ecce6"
server_key_version: "encryptionkey@0"
I20260925 11:11:37.047652 27516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "e16ccfcede2e47c5808a92231c9ee564"
format_stamp: "Formatted at 2026-09-25 11:11:37 on dist-test-slave-6042"
server_key: "190959602705eda3fdd7de282c88cdb4"
server_key_iv: "a7887604e41a15137c8960efb33ecce6"
server_key_version: "encryptionkey@0"
I20260925 11:11:37.053066 27516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:37.056711 27531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:37.058130 27516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:37.058310 27516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "e16ccfcede2e47c5808a92231c9ee564"
format_stamp: "Formatted at 2026-09-25 11:11:37 on dist-test-slave-6042"
server_key: "190959602705eda3fdd7de282c88cdb4"
server_key_iv: "a7887604e41a15137c8960efb33ecce6"
server_key_version: "encryptionkey@0"
I20260925 11:11:37.058611 27516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:37.076429 27516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:37.080595 27516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:37.080850 27516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:37.081789 27516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:37.083206 27516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:37.083279 27516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:37.083365 27516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:37.083410 27516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:37.102974 27376 heartbeater.cc:503] Master 127.8.181.62:35285 was elected leader, sending a full tablet report...
I20260925 11:11:37.118166 27516 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:46847
I20260925 11:11:37.118189 27643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:46847 every 8 connection(s)
I20260925 11:11:37.119498 27516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:37.124878 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27516
I20260925 11:11:37.125057 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:37.125613 8916 external_mini_cluster.cc:1428] Setting key 3323734a0d2fc789d7fdf40206a2e79e
Sep 25 11:11:37 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334697, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:37.141036 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.122054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:38317 (local address 127.8.181.62:35285)
0925 11:11:37.122584 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.122590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.123447 (+ 857us) server_negotiation.cc:407] Connection header received
0925 11:11:37.125773 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.125778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.125862 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.125957 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:37.128028 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.129127 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.130800 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.131246 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.131352 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.134828 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:37.134833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:37.134837 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:37.134875 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:37.137474 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.138337 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.138341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.138343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.138555 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.138971 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.138974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.138976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.139348 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:37.139498 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.140399 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.140607 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":253,"threads_started":1}
I20260925 11:11:37.142004 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.122446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35285 (local address 127.8.181.3:38317)
0925 11:11:37.123264 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.123305 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.125437 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.126214 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.126223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.126993 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:37.127799 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.127818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.129295 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.129300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.130625 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.130637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.131451 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.131455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.131646 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.132331 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:37.132359 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:37.134600 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:37.137609 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:37.137615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:37.137628 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:37.138145 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:37.138670 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:37.138673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:37.138676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:37.138851 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:37.139486 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:37.139494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:37.139959 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.140277 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.140783 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":276,"threads_started":1}
I20260925 11:11:37.143492 27644 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35285
I20260925 11:11:37.143812 27644 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:37.144594 27644 heartbeater.cc:511] Master 127.8.181.62:35285 requested a full tablet report, sending...
I20260925 11:11:37.146018 27186 ts_manager.cc:194] Registered new tserver with Master: e16ccfcede2e47c5808a92231c9ee564 (127.8.181.3:46847)
I20260925 11:11:37.146875 27186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:38317
I20260925 11:11:37.153430 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:37.171445 27646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.157703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.8.181.62:35285)
0925 11:11:37.157890 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.157895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.158510 (+ 615us) server_negotiation.cc:407] Connection header received
0925 11:11:37.158743 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.158747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.158815 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.158919 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:37.160323 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.161075 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.162458 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.162932 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.163062 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.165737 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:37.165743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:37.165746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:37.165781 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:37.168498 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.169333 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.169338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.169346 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.169478 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.169931 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.169936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.169943 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.170246 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:37.170566 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.170948 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.171143 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:11:37.176043 27186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:37.183139 27186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:37.206071 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.196177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:44817 (local address 127.0.0.1:59990)
0925 11:11:37.197265 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.197278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.197408 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.198154 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.198161 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.198206 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:37.198748 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.198761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.200915 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.200927 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.203501 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.203515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.204939 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.204945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.205190 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.205228 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.205384 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.205579 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":31,"thread_start_us":1861,"threads_started":1}
I20260925 11:11:37.207101 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.197246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.8.181.3:46847)
0925 11:11:37.198020 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.198029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.198429 (+ 400us) server_negotiation.cc:407] Connection header received
0925 11:11:37.198645 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.198654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.198883 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.199072 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:37.199926 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.201457 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.203889 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.205311 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.205469 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.205696 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.206567 (+ 871us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.206742 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":480,"threads_started":1}
I20260925 11:11:37.207331 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.196440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.8.181.2:44817)
0925 11:11:37.197090 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.197100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.197372 (+ 272us) server_negotiation.cc:407] Connection header received
0925 11:11:37.197569 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.197579 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.197877 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.198134 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:37.198945 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.200725 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.203682 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.204725 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.204880 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.205096 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.206823 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.206956 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":334,"threads_started":1}
I20260925 11:11:37.207710 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.196638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46847 (local address 127.0.0.1:55842)
0925 11:11:37.198303 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.198322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.198455 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.199342 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.199347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.199371 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:37.199725 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.199737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.201627 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.201636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.203715 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.203731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.205845 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.205851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.206064 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.206097 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.206192 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.206321 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1539,"mutex_wait_us":27,"thread_start_us":912,"threads_started":1}
I20260925 11:11:37.208141 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.195481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:36481 (local address 127.0.0.1:43190)
0925 11:11:37.196178 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.196226 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.196409 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.200500 (+ 4091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.200504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.200526 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:37.201021 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.201059 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.203077 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.203083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.205290 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.205305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.207195 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.207201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.207395 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.207443 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.207555 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.207677 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":6,"thread_start_us":298,"threads_started":1}
I20260925 11:11:37.209205 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.195712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.8.181.1:36481)
0925 11:11:37.200042 (+ 4330us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.200051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.200095 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:11:37.200177 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.200182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.200326 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.200502 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:37.201238 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.202930 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.205492 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.206952 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.207095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.207375 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.207597 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.207782 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4125,"thread_start_us":3971,"threads_started":1}
I20260925 11:11:37.211457 27445 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e3924a7a40cc80e010852dddf7ea (DEFAULT_TABLE table=test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:37.212673 27445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527e3924a7a40cc80e010852dddf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:37.212782 27579 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e3924a7a40cc80e010852dddf7ea (DEFAULT_TABLE table=test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:37.212782 27311 tablet_service.cc:1467] Processing CreateTablet for tablet 7527e3924a7a40cc80e010852dddf7ea (DEFAULT_TABLE table=test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:37.214133 27579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527e3924a7a40cc80e010852dddf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:37.214138 27311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7527e3924a7a40cc80e010852dddf7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:37.227514 27664 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Bootstrap starting.
I20260925 11:11:37.229594 27663 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Bootstrap starting.
I20260925 11:11:37.232385 27665 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Bootstrap starting.
I20260925 11:11:37.232599 27664 tablet_bootstrap.cc:654] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:37.233242 27663 tablet_bootstrap.cc:654] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:37.234025 27664 log.cc:824] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:37.234220 27663 log.cc:824] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:37.236056 27665 tablet_bootstrap.cc:654] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:37.237090 27665 log.cc:824] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:37.237363 27664 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: No bootstrap required, opened a new log
I20260925 11:11:37.237365 27663 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: No bootstrap required, opened a new log
I20260925 11:11:37.237656 27664 ts_tablet_manager.cc:1397] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 11:11:37.237655 27663 ts_tablet_manager.cc:1397] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 11:11:37.244892 27665 tablet_bootstrap.cc:492] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: No bootstrap required, opened a new log
I20260925 11:11:37.245196 27665 ts_tablet_manager.cc:1397] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 11:11:37.248683 27664 raft_consensus.cc:348] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.248683 27663 raft_consensus.cc:348] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.249164 27663 raft_consensus.cc:374] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:37.249166 27664 raft_consensus.cc:374] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:37.249266 27664 raft_consensus.cc:729] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ecb268bf3441248a2176e1b5a15a2c, State: Initialized, Role: FOLLOWER
I20260925 11:11:37.249327 27663 raft_consensus.cc:729] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a32f9df41e64260b70d260711405c88, State: Initialized, Role: FOLLOWER
I20260925 11:11:37.249964 27663 consensus_queue.cc:260] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.249976 27664 consensus_queue.cc:260] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.252780 27664 ts_tablet_manager.cc:1428] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 11:11:37.252873 27663 ts_tablet_manager.cc:1428] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260925 11:11:37.253356 27510 heartbeater.cc:503] Master 127.8.181.62:35285 was elected leader, sending a full tablet report...
I20260925 11:11:37.253603 27671 raft_consensus.cc:482] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:37.253878 27671 raft_consensus.cc:504] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.254173 27665 raft_consensus.cc:348] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.254763 27665 raft_consensus.cc:374] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:37.254937 27665 raft_consensus.cc:729] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16ccfcede2e47c5808a92231c9ee564, State: Initialized, Role: FOLLOWER
I20260925 11:11:37.255626 27665 consensus_queue.cc:260] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.262887 27644 heartbeater.cc:503] Master 127.8.181.62:35285 was elected leader, sending a full tablet report...
I20260925 11:11:37.264034 27665 ts_tablet_manager.cc:1428] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260925 11:11:37.266103 27671 leader_election.cc:288] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5ecb268bf3441248a2176e1b5a15a2c (127.8.181.1:36481), e16ccfcede2e47c5808a92231c9ee564 (127.8.181.3:46847)
I20260925 11:11:37.266716 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.256695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:43261 (local address 127.8.181.1:36481)
0925 11:11:37.256872 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.256880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.256943 (+ 63us) server_negotiation.cc:407] Connection header received
0925 11:11:37.257649 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.257655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.257738 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.257808 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:37.258648 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.260206 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.263663 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.265146 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.265290 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.265441 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.266213 (+ 772us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.266373 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 11:11:37.267634 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.256337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:36481 (local address 127.8.181.2:43261)
0925 11:11:37.256843 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.256863 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.257057 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.258020 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.258024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.258052 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:37.258431 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.258443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.260436 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.260444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.263478 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.263493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.265715 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.265725 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.265994 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.266042 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.266908 (+ 866us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.267132 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":559,"threads_started":1}
I20260925 11:11:37.268981 27331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e3924a7a40cc80e010852dddf7ea" candidate_uuid: "2a32f9df41e64260b70d260711405c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" is_pre_election: true
I20260925 11:11:37.269498 27331 raft_consensus.cc:2454] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a32f9df41e64260b70d260711405c88 in term 0.
I20260925 11:11:37.270169 27398 leader_election.cc:302] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a32f9df41e64260b70d260711405c88, b5ecb268bf3441248a2176e1b5a15a2c; no voters:
I20260925 11:11:37.270593 27671 raft_consensus.cc:2781] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:37.270752 27671 raft_consensus.cc:482] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:37.270852 27671 raft_consensus.cc:3037] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:37.273447 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.258510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:42221 (local address 127.8.181.3:46847)
0925 11:11:37.258720 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.258726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.268061 (+ 9335us) server_negotiation.cc:407] Connection header received
0925 11:11:37.268323 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.268327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.268384 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.268495 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:37.268828 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.269616 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.271664 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.272476 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.272585 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.272662 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.273132 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.273227 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 11:11:37.273627 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.257960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:46847 (local address 127.8.181.2:42221)
0925 11:11:37.267946 (+ 9986us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:37.267971 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:37.268163 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:37.268509 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:37.268513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:37.268541 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:37.268702 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.268710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.269788 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.269794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.271488 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:37.271504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.272688 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.272695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:37.272946 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.273000 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 11:11:37.273111 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 11:11:37.273269 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9809,"thread_start_us":10096,"threads_started":1}
I20260925 11:11:37.274098 27671 raft_consensus.cc:504] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.275032 27671 leader_election.cc:288] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [CANDIDATE]: Term 1 election: Requested vote from peers b5ecb268bf3441248a2176e1b5a15a2c (127.8.181.1:36481), e16ccfcede2e47c5808a92231c9ee564 (127.8.181.3:46847)
I20260925 11:11:37.274864 27599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e3924a7a40cc80e010852dddf7ea" candidate_uuid: "2a32f9df41e64260b70d260711405c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16ccfcede2e47c5808a92231c9ee564" is_pre_election: true
I20260925 11:11:37.275426 27331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e3924a7a40cc80e010852dddf7ea" candidate_uuid: "2a32f9df41e64260b70d260711405c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5ecb268bf3441248a2176e1b5a15a2c"
I20260925 11:11:37.275594 27331 raft_consensus.cc:3037] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:37.275576 27599 raft_consensus.cc:2454] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a32f9df41e64260b70d260711405c88 in term 0.
I20260925 11:11:37.275537 27598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7527e3924a7a40cc80e010852dddf7ea" candidate_uuid: "2a32f9df41e64260b70d260711405c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16ccfcede2e47c5808a92231c9ee564"
I20260925 11:11:37.275811 27598 raft_consensus.cc:3037] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:37.277853 27331 raft_consensus.cc:2454] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a32f9df41e64260b70d260711405c88 in term 1.
I20260925 11:11:37.278261 27398 leader_election.cc:302] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a32f9df41e64260b70d260711405c88, b5ecb268bf3441248a2176e1b5a15a2c; no voters:
I20260925 11:11:37.278575 27671 raft_consensus.cc:2781] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:37.278669 27598 raft_consensus.cc:2454] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a32f9df41e64260b70d260711405c88 in term 1.
I20260925 11:11:37.278797 27671 raft_consensus.cc:686] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 1 LEADER]: Becoming Leader. State: Replica: 2a32f9df41e64260b70d260711405c88, State: Running, Role: LEADER
I20260925 11:11:37.279317 27671 consensus_queue.cc:237] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } }
I20260925 11:11:37.283826 27186 catalog_manager.cc:5495] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a32f9df41e64260b70d260711405c88 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "2a32f9df41e64260b70d260711405c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a32f9df41e64260b70d260711405c88" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 } health_report { overall_health: UNKNOWN } } }
I20260925 11:11:37.313314 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.308224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.8.181.2:44817)
0925 11:11:37.308468 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.308475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.308494 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:11:37.308637 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.308641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.308719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.308844 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:37.309190 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.309868 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.311257 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.311661 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.311758 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.312216 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:37.312703 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:37.312829 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.312945 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.313043 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:11:37.321337 27331 raft_consensus.cc:1261] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 1 FOLLOWER]: Refusing update from remote peer 2a32f9df41e64260b70d260711405c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:37.321337 27598 raft_consensus.cc:1261] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 1 FOLLOWER]: Refusing update from remote peer 2a32f9df41e64260b70d260711405c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:37.322273 27669 consensus_queue.cc:1035] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5ecb268bf3441248a2176e1b5a15a2c" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:37.322719 27671 consensus_queue.cc:1035] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16ccfcede2e47c5808a92231c9ee564" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 11:11:37.322769 27377 tablet.cc:2369] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:37.336926 27678 mvcc.cc:205] Tried to move back new op lower bound from 7333210920219013120 to 7333210920060366848. Current Snapshot: MvccSnapshot[applied={T|T < 7333210920219013120}]
I20260925 11:11:37.347846 27184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:37.348119 27184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:37.351787 27184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 844ff74037944fcd812bb7c408dabb37: Sending DeleteTablet for 3 replicas of tablet 7527e3924a7a40cc80e010852dddf7ea
I20260925 11:11:37.352924 27445 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e3924a7a40cc80e010852dddf7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:59990
I20260925 11:11:37.353340 27311 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e3924a7a40cc80e010852dddf7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:43190
I20260925 11:11:37.354036 27579 tablet_service.cc:1514] Processing DeleteTablet for tablet 7527e3924a7a40cc80e010852dddf7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:37 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:55842
I20260925 11:11:37.354403 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27248
I20260925 11:11:37.377725 27689 tablet_replica.cc:330] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: stopping tablet replica
I20260925 11:11:37.378515 27688 tablet_replica.cc:330] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: stopping tablet replica
I20260925 11:11:37.379150 27689 raft_consensus.cc:2229] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:37.379724 27688 raft_consensus.cc:2229] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:37.380285 27688 raft_consensus.cc:2258] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:37.380268 27689 raft_consensus.cc:2258] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:37.382615 27689 ts_tablet_manager.cc:1905] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:37.382616 27688 ts_tablet_manager.cc:1905] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 11:11:37 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:11:37.389837 27688 ts_tablet_manager.cc:1918] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:37.389998 27689 ts_tablet_manager.cc:1918] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:37.390048 27688 log.cc:1196] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/7527e3924a7a40cc80e010852dddf7ea
I20260925 11:11:37.390152 27689 log.cc:1196] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/7527e3924a7a40cc80e010852dddf7ea
I20260925 11:11:37.390656 27688 ts_tablet_manager.cc:1939] T 7527e3924a7a40cc80e010852dddf7ea P b5ecb268bf3441248a2176e1b5a15a2c: Deleting consensus metadata
I20260925 11:11:37.390753 27689 ts_tablet_manager.cc:1939] T 7527e3924a7a40cc80e010852dddf7ea P e16ccfcede2e47c5808a92231c9ee564: Deleting consensus metadata
I20260925 11:11:37.392805 27170 catalog_manager.cc:4841] TS e16ccfcede2e47c5808a92231c9ee564 (127.8.181.3:46847): tablet 7527e3924a7a40cc80e010852dddf7ea (table test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]) successfully deleted
I20260925 11:11:37.392937 27687 tablet_replica.cc:330] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: stopping tablet replica
I20260925 11:11:37.393214 27170 catalog_manager.cc:4841] TS b5ecb268bf3441248a2176e1b5a15a2c (127.8.181.1:36481): tablet 7527e3924a7a40cc80e010852dddf7ea (table test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]) successfully deleted
I20260925 11:11:37.393424 27687 raft_consensus.cc:2229] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:37.394151 27687 raft_consensus.cc:2258] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:37.396227 27687 ts_tablet_manager.cc:1905] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:37.397291 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.356185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.8.181.1:36481)
0925 11:11:37.356431 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.356440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.368166 (+ 11726us) server_negotiation.cc:407] Connection header received
0925 11:11:37.368801 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.368811 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.368968 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.369074 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:37.373076 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.374604 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.377228 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.378149 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.378327 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.385414 (+ 7087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:37.385435 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:37.385461 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:37.385519 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:37.393667 (+ 8148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.394777 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.394787 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.394791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.394957 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.395449 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.395454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.395458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.395914 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:37.396200 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.396626 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.396856 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 11:11:37.398057 27371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:37.406023 27687 ts_tablet_manager.cc:1918] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:37.406306 27687 log.cc:1196] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/7527e3924a7a40cc80e010852dddf7ea
I20260925 11:11:37.406996 27687 ts_tablet_manager.cc:1939] T 7527e3924a7a40cc80e010852dddf7ea P 2a32f9df41e64260b70d260711405c88: Deleting consensus metadata
I20260925 11:11:37.409124 27171 catalog_manager.cc:4841] TS 2a32f9df41e64260b70d260711405c88 (127.8.181.2:44817): tablet 7527e3924a7a40cc80e010852dddf7ea (table test-table [id=b4438c27ef29486f9d1e7a5b1898fac2]) successfully deleted
I20260925 11:11:37.511971 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27248
I20260925 11:11:37.531705 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27382
Sep 25 11:11:37 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:11:37.549856 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.532627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.8.181.2:44817)
0925 11:11:37.532828 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.532835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.532858 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:11:37.532953 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.532959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.533039 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.533160 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:37.534593 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.535489 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.536966 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.537474 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.537640 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.542606 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:37.542628 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:37.542651 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:37.542727 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:37.546256 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.547293 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.547301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.547304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.547478 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.547966 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.547973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.547976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.548389 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:37.548788 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.549159 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.549416 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 11:11:37.550753 27505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:37.666801 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27382
I20260925 11:11:37.685986 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27516
Sep 25 11:11:37 dist-test-slave-6042 krb5kdc[27141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:11:37.701818 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:37.686787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.8.181.3:46847)
0925 11:11:37.686990 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:37.686995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:37.687051 (+ 56us) server_negotiation.cc:407] Connection header received
0925 11:11:37.687200 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:37.687204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:37.687269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:37.687366 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:37.688783 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.689536 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.691215 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:37.691720 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:37.691820 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:37.696142 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:37.696158 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:37.696173 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:37.696208 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:37.699049 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.699865 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.699872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.699874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.700027 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:37.700391 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:37.700394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:37.700396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:37.700760 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:37.701040 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:37.701315 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 11:11:37.701497 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 11:11:37.702581 27639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:37.795055 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27516
I20260925 11:11:37.812651 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27154
I20260925 11:11:37.813735 27216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:37.925060 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27154
2026-09-25T11:11:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4361 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T11:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:37Z Disabled control of system clock
I20260925 11:11:38.003229 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38203
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:38203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:38.338824 27698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:38.339125 27698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:38.339205 27698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:38.346377 27698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:38.346522 27698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:38.346562 27698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:38.346635 27698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:38.346680 27698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:38.355964 27698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:38203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:38203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:38.357266 27698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:38.359273 27698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:38.370991 27704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:38.371215 27703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:38.371484 27706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:38.372200 27698 server_base.cc:1034] running on GCE node
I20260925 11:11:38.373077 27698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:38.374591 27698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:38.375829 27698 hybrid_clock.cc:648] HybridClock initialized: now 1790334698375774 us; error 61 us; skew 500 ppm
I20260925 11:11:38.376187 27698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:38.378552 27698 webserver.cc:460] Webserver started at http://127.8.181.62:39705/ using document root <none> and password file <none>
I20260925 11:11:38.379289 27698 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:38.379415 27698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:38.379743 27698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:38.382802 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "a647ded6b0ba4f57a4aff012ccac678e"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "63343f5c454f411ff136efd7fb6a71af"
server_key_iv: "100566ff78d94c63441683ab0d922b3f"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.383621 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "a647ded6b0ba4f57a4aff012ccac678e"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "63343f5c454f411ff136efd7fb6a71af"
server_key_iv: "100566ff78d94c63441683ab0d922b3f"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.388816 27698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:38.392421 27712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:38.393841 27698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:38.394048 27698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a647ded6b0ba4f57a4aff012ccac678e"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "63343f5c454f411ff136efd7fb6a71af"
server_key_iv: "100566ff78d94c63441683ab0d922b3f"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.394280 27698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:38.407147 27698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:38.408250 27698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:38.408753 27698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:38.427909 27698 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:38203
I20260925 11:11:38.427932 27763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:38203 every 8 connection(s)
I20260925 11:11:38.429375 27698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:38.433238 27764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:38.439026 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27698
I20260925 11:11:38.439245 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:38.439702 8916 external_mini_cluster.cc:1428] Setting key 491e15766f656b35db1cc5fdd1405b85
I20260925 11:11:38.445508 27764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: Bootstrap starting.
I20260925 11:11:38.449568 27764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:38.450728 27764 log.cc:824] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:38.453701 27764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: No bootstrap required, opened a new log
I20260925 11:11:38.455468 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:38.441758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.8.181.62:38203)
0925 11:11:38.442919 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:38.442934 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:38.443001 (+ 67us) server_negotiation.cc:407] Connection header received
0925 11:11:38.445415 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:38.445444 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:38.446064 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:38.446534 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:38.447091 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:38.449118 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:38.450606 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:38.451671 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:38.451884 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:38.453122 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:38.453129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:38.453143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:38.453704 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:38.453783 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:38.453829 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:38.453940 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:38.454161 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 11:11:38.454515 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":324,"threads_started":1}
I20260925 11:11:38.462270 27764 raft_consensus.cc:348] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } }
I20260925 11:11:38.462723 27764 raft_consensus.cc:374] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:38.462826 27764 raft_consensus.cc:729] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a647ded6b0ba4f57a4aff012ccac678e, State: Initialized, Role: FOLLOWER
I20260925 11:11:38.463382 27764 consensus_queue.cc:260] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } }
I20260925 11:11:38.463631 27764 raft_consensus.cc:388] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:38.463778 27764 raft_consensus.cc:482] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:38.463958 27764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:38.466146 27764 raft_consensus.cc:504] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } }
I20260925 11:11:38.466743 27764 leader_election.cc:302] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a647ded6b0ba4f57a4aff012ccac678e; no voters:
I20260925 11:11:38.467406 27764 leader_election.cc:288] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:38.467566 27769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:38.468483 27769 raft_consensus.cc:686] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [term 1 LEADER]: Becoming Leader. State: Replica: a647ded6b0ba4f57a4aff012ccac678e, State: Running, Role: LEADER
I20260925 11:11:38.469120 27769 consensus_queue.cc:237] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } }
I20260925 11:11:38.469878 27764 sys_catalog.cc:564] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:38.472035 27771 sys_catalog.cc:455] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a647ded6b0ba4f57a4aff012ccac678e. Latest consensus state: current_term: 1 leader_uuid: "a647ded6b0ba4f57a4aff012ccac678e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } } }
I20260925 11:11:38.472941 27771 sys_catalog.cc:458] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:38.472635 27770 sys_catalog.cc:455] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a647ded6b0ba4f57a4aff012ccac678e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a647ded6b0ba4f57a4aff012ccac678e" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 38203 } } }
I20260925 11:11:38.473479 27770 sys_catalog.cc:458] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:38.474435 27776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:38.480216 27776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:38.490084 27776 catalog_manager.cc:1347] Generated new cluster ID: bd6ab69ba50044fd8081aebb37c46df5
I20260925 11:11:38.490204 27776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:38.508955 27776 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:38.510347 27776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:38.519259 27776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: Generated new TSK 0
I20260925 11:11:38.520148 27776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:38.532840 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--builtin_ntp_servers=127.8.181.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:38.890673 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:38.891017 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:38.891171 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:38.898892 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:38.899047 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:38.899227 27788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:38.908604 27788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:38.909937 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:38.911767 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:38.926357 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:38.926286 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:38.926324 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:38.927452 27788 server_base.cc:1034] running on GCE node
I20260925 11:11:38.928144 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:38.929226 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:38.930507 27788 hybrid_clock.cc:648] HybridClock initialized: now 1790334698930410 us; error 54 us; skew 500 ppm
I20260925 11:11:38.930886 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:38.933338 27788 webserver.cc:460] Webserver started at http://127.8.181.1:42259/ using document root <none> and password file <none>
I20260925 11:11:38.934069 27788 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:38.934199 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:38.934574 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:38.937489 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "6f69ee4bc1824f45a762b762edb52558"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "4b34379c9c3e1124760f7b2c6c89f173"
server_key_iv: "9ac2ca12f71f47c5c7743a0f0ed46a3b"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.938333 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "6f69ee4bc1824f45a762b762edb52558"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "4b34379c9c3e1124760f7b2c6c89f173"
server_key_iv: "9ac2ca12f71f47c5c7743a0f0ed46a3b"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.943818 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 11:11:38.947526 27802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:38.948902 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:38.949102 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "6f69ee4bc1824f45a762b762edb52558"
format_stamp: "Formatted at 2026-09-25 11:11:38 on dist-test-slave-6042"
server_key: "4b34379c9c3e1124760f7b2c6c89f173"
server_key_iv: "9ac2ca12f71f47c5c7743a0f0ed46a3b"
server_key_version: "encryptionkey@0"
I20260925 11:11:38.949352 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:38.968909 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:38.969826 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:38.970155 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:38.971315 27788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:38.972914 27788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:38.973031 27788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:38.973160 27788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:38.973248 27788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.007241 27788 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:42725
I20260925 11:11:39.007328 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:42725 every 8 connection(s)
I20260925 11:11:39.008574 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:39.013180 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27788
I20260925 11:11:39.013355 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:39.013957 8916 external_mini_cluster.cc:1428] Setting key 611e1db6b6143b0e5c25510646a3db59
I20260925 11:11:39.017257 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--builtin_ntp_servers=127.8.181.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:39.023947 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.011144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:50771 (local address 127.8.181.62:38203)
0925 11:11:39.011612 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:39.011616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:39.012659 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 11:11:39.015455 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:39.015462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:39.015556 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:39.015673 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:39.017071 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.018098 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.020692 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.021104 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.021212 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.023069 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:39.023075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:39.023079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:39.023101 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:39.023148 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:39.023161 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:39.023250 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:39.023579 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 11:11:39.023668 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":250,"threads_started":1}
I20260925 11:11:39.025359 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.011587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38203 (local address 127.8.181.1:50771)
0925 11:11:39.012485 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:39.012533 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:39.015136 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:39.015928 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:39.015937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:39.016033 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:39.016857 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.016883 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.018283 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.018292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.020517 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.020535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.021342 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.021352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.021703 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.022541 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:39.022547 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:39.022605 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:39.022724 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:39.023278 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:39.023292 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:39.023303 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 11:11:39.023500 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 11:11:39.023953 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":271,"threads_started":1}
I20260925 11:11:39.027066 27915 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38203
I20260925 11:11:39.027396 27915 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:39.028167 27915 heartbeater.cc:511] Master 127.8.181.62:38203 requested a full tablet report, sending...
I20260925 11:11:39.030177 27729 ts_manager.cc:194] Registered new tserver with Master: 6f69ee4bc1824f45a762b762edb52558 (127.8.181.1:42725)
I20260925 11:11:39.031693 27729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.1:50771
W20260925 11:11:39.351166 27919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:39.351462 27919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:39.351624 27919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:39.359215 27919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:39.359331 27919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:39.359509 27919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:39.369088 27919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:39.370414 27919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:39.372263 27919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:39.386018 27924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:39.386088 27925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:39.386051 27927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:39.386888 27919 server_base.cc:1034] running on GCE node
I20260925 11:11:39.387478 27919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:39.388414 27919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:39.389629 27919 hybrid_clock.cc:648] HybridClock initialized: now 1790334699389577 us; error 42 us; skew 500 ppm
I20260925 11:11:39.389969 27919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:39.392632 27919 webserver.cc:460] Webserver started at http://127.8.181.2:38203/ using document root <none> and password file <none>
I20260925 11:11:39.393337 27919 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:39.393427 27919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:39.393826 27919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:39.396879 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "bc4f25be019448089bf91fb9df83ad99"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "9f9495b74f13a6a24be6688d1ed1db9e"
server_key_iv: "f274ea448d20afb0c17ea0f493982fc2"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.397728 27919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "bc4f25be019448089bf91fb9df83ad99"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "9f9495b74f13a6a24be6688d1ed1db9e"
server_key_iv: "f274ea448d20afb0c17ea0f493982fc2"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.403191 27919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 11:11:39.406992 27933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.408411 27919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:11:39.408622 27919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "bc4f25be019448089bf91fb9df83ad99"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "9f9495b74f13a6a24be6688d1ed1db9e"
server_key_iv: "f274ea448d20afb0c17ea0f493982fc2"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.408838 27919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:39.442876 27919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:39.443904 27919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:39.444208 27919 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260925 11:11:39.444619 27760 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.8.181.62:38203
I20260925 11:11:39.445441 27919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:39.447391 27919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:39.447517 27919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.447628 27919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:39.447695 27919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.481802 27919 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:43987
I20260925 11:11:39.481894 28045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:43987 every 8 connection(s)
I20260925 11:11:39.483147 27919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:39.485846 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 27919
I20260925 11:11:39.486035 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:39.486521 8916 external_mini_cluster.cc:1428] Setting key b5bebf9d65398c8861cc42a734fbf1b4
I20260925 11:11:39.489717 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--builtin_ntp_servers=127.8.181.20:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:39.499342 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.485728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:37393 (local address 127.8.181.62:38203)
0925 11:11:39.485970 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:39.485976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:39.487674 (+ 1698us) server_negotiation.cc:407] Connection header received
0925 11:11:39.489981 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:39.489989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:39.490097 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:39.490172 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:39.491925 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.492650 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.495147 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.495868 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.496104 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.497932 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:39.497975 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:39.497983 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:39.498016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:39.498091 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:39.498140 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:39.498268 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:39.498631 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 11:11:39.498810 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:11:39.500722 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.486403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38203 (local address 127.8.181.2:37393)
0925 11:11:39.487504 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:39.487554 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:39.489635 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:39.490739 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:39.490754 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:39.490894 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:39.491725 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.491753 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.492811 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.492819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.494970 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.494989 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.496129 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.496138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.496483 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.497379 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:39.497385 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:39.497441 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:39.497617 (+ 176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:39.498266 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:39.498280 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:39.498292 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 11:11:39.498620 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 11:11:39.499130 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":345,"threads_started":1}
I20260925 11:11:39.502604 28046 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38203
I20260925 11:11:39.503010 28046 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:39.504072 28046 heartbeater.cc:511] Master 127.8.181.62:38203 requested a full tablet report, sending...
I20260925 11:11:39.505653 27729 ts_manager.cc:194] Registered new tserver with Master: bc4f25be019448089bf91fb9df83ad99 (127.8.181.2:43987)
I20260925 11:11:39.506520 27729 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.2:37393
W20260925 11:11:39.831557 28049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:39.831905 28049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:39.832064 28049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:39.839525 28049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:39.839668 28049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:39.839843 28049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:39.849272 28049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:38203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:39.850658 28049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:39.852532 28049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:39.865814 28054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:39.865882 28057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:39.865841 28055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:39.866731 28049 server_base.cc:1034] running on GCE node
I20260925 11:11:39.867384 28049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:39.868338 28049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:39.869570 28049 hybrid_clock.cc:648] HybridClock initialized: now 1790334699869525 us; error 56 us; skew 500 ppm
I20260925 11:11:39.869890 28049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:39.872575 28049 webserver.cc:460] Webserver started at http://127.8.181.3:34391/ using document root <none> and password file <none>
I20260925 11:11:39.873268 28049 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:39.873389 28049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:39.873708 28049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:39.876720 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "607c2e4fc1a14faa933d2a078e17ef4e"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "8e7e952d5fc41f529ce72a6d2e1da8d7"
server_key_iv: "451cbc42348a677a6ba34f3ef795055c"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.877458 28049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "607c2e4fc1a14faa933d2a078e17ef4e"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "8e7e952d5fc41f529ce72a6d2e1da8d7"
server_key_iv: "451cbc42348a677a6ba34f3ef795055c"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.882815 28049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 11:11:39.886668 28063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.888036 28049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:39.888180 28049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "607c2e4fc1a14faa933d2a078e17ef4e"
format_stamp: "Formatted at 2026-09-25 11:11:39 on dist-test-slave-6042"
server_key: "8e7e952d5fc41f529ce72a6d2e1da8d7"
server_key_iv: "451cbc42348a677a6ba34f3ef795055c"
server_key_version: "encryptionkey@0"
I20260925 11:11:39.888376 28049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:39.918205 28049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:39.919164 28049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:39.919425 28049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:39.920550 28049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:39.922335 28049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:39.922435 28049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.922590 28049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:39.922655 28049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:39.957212 28049 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:45319
I20260925 11:11:39.957269 28175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:45319 every 8 connection(s)
I20260925 11:11:39.958564 28049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:39.960755 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28049
I20260925 11:11:39.960958 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:39.961426 8916 external_mini_cluster.cc:1428] Setting key a454bf0775ee3578b6cd0047043782fd
I20260925 11:11:39.971602 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.961066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:52137 (local address 127.8.181.62:38203)
0925 11:11:39.961288 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:39.961294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:39.962843 (+ 1549us) server_negotiation.cc:407] Connection header received
0925 11:11:39.965145 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:39.965151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:39.965238 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:39.965349 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:39.966417 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.967231 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.968790 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.969194 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.969323 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.970759 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:39.970765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:39.970770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:39.970795 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:39.970827 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:39.970838 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:39.970940 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:39.971175 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 11:11:39.971274 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:11:39.972575 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.961645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:38203 (local address 127.8.181.3:52137)
0925 11:11:39.962702 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:39.962748 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:39.964883 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:39.965583 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:39.965591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:39.965679 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:39.966252 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.966269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.967392 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.967396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.968648 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:39.968660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.969416 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.969423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:39.969655 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.970366 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:39.970369 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:39.970411 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:39.970579 (+ 168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:39.970950 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:39.970961 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:39.970969 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 11:11:39.971156 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 11:11:39.971549 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":344,"threads_started":1}
I20260925 11:11:39.973922 28176 heartbeater.cc:347] Connected to a master server at 127.8.181.62:38203
I20260925 11:11:39.974275 28176 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:39.975186 28176 heartbeater.cc:511] Master 127.8.181.62:38203 requested a full tablet report, sending...
I20260925 11:11:39.976644 27728 ts_manager.cc:194] Registered new tserver with Master: 607c2e4fc1a14faa933d2a078e17ef4e (127.8.181.3:45319)
I20260925 11:11:39.977505 27728 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.3:52137
I20260925 11:11:39.989089 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:39.999919 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:39.993357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.8.181.62:38203)
0925 11:11:39.993574 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:39.993581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:39.994150 (+ 569us) server_negotiation.cc:407] Connection header received
0925 11:11:39.994379 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:39.994385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:39.994540 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:39.994650 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:39.995146 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.995854 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.997234 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:39.997668 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:39.997833 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:39.999133 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:39.999139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:39.999144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:39.999169 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:39.999200 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:39.999212 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:39.999361 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:39.999550 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 11:11:39.999607 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 11:11:40.004202 27728 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:40.013682 27728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:40.034776 27915 heartbeater.cc:503] Master 127.8.181.62:38203 was elected leader, sending a full tablet report...
I20260925 11:11:40.037616 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.026168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.8.181.1:42725)
0925 11:11:40.027824 (+ 1656us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.027832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.028537 (+ 705us) server_negotiation.cc:407] Connection header received
0925 11:11:40.028860 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.028867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.029092 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.029259 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:40.030179 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.031786 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.034293 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.035770 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.035874 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.036178 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.037072 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.037256 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1484,"thread_start_us":1664,"threads_started":1}
I20260925 11:11:40.037662 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.025542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:43987 (local address 127.0.0.1:33728)
0925 11:11:40.026681 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.026732 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:40.026953 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:40.028200 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:40.028210 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:40.028252 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:40.028763 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.028779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.031000 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.031005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.033593 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.033608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.036823 (+ 3215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.036830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.037018 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.037046 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 11:11:40.037102 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 11:11:40.037296 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":4,"thread_start_us":2567,"threads_started":1}
I20260925 11:11:40.038436 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.025739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47054)
0925 11:11:40.028426 (+ 2687us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.028447 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:40.028654 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:40.029533 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:40.029537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:40.029568 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:40.029978 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.029989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.031977 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.031982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.034085 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.034097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.036513 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.036520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.036765 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.036840 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 11:11:40.037879 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0925 11:11:40.038038 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2547,"mutex_wait_us":1,"thread_start_us":618,"threads_started":1}
I20260925 11:11:40.039119 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.025829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.8.181.2:43987)
0925 11:11:40.027532 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.027544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.027576 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:11:40.027682 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.027690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.027938 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.028968 (+ 1030us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:40.029191 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.030831 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.035206 (+ 4375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.036609 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.038159 (+ 1550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.038426 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.038575 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.038774 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":1422,"threads_started":1}
I20260925 11:11:40.042923 27850 tablet_service.cc:1467] Processing CreateTablet for tablet 488ba0c4b011452ea358ffa1960fad75 (DEFAULT_TABLE table=test-table [id=67ddc8a371d745ecb1bdfcca0b6035fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:40.043543 27981 tablet_service.cc:1467] Processing CreateTablet for tablet 488ba0c4b011452ea358ffa1960fad75 (DEFAULT_TABLE table=test-table [id=67ddc8a371d745ecb1bdfcca0b6035fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:40.044422 27850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488ba0c4b011452ea358ffa1960fad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:40.044708 27981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488ba0c4b011452ea358ffa1960fad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:40.047441 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.028606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45319 (local address 127.0.0.1:58236)
0925 11:11:40.037879 (+ 9273us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.037899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:40.037995 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:40.040717 (+ 2722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:40.040723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:40.040752 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:40.040926 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.040940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.043040 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.043045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.044813 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.044828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.046629 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.046636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.046824 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.046863 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 11:11:40.046966 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 11:11:40.047127 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9170,"thread_start_us":10813,"threads_started":1}
I20260925 11:11:40.048491 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.028972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.8.181.3:45319)
0925 11:11:40.040059 (+ 11087us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.040071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.040108 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:11:40.040223 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.040233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.040485 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.041113 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:40.041382 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.042865 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.044975 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.046390 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.046569 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.046834 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.047026 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.048126 (+ 1100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10906,"thread_start_us":10730,"threads_started":1}
I20260925 11:11:40.052614 28111 tablet_service.cc:1467] Processing CreateTablet for tablet 488ba0c4b011452ea358ffa1960fad75 (DEFAULT_TABLE table=test-table [id=67ddc8a371d745ecb1bdfcca0b6035fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:40.055074 28111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488ba0c4b011452ea358ffa1960fad75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:40.057027 28194 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: Bootstrap starting.
I20260925 11:11:40.059084 28195 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Bootstrap starting.
I20260925 11:11:40.060920 28194 tablet_bootstrap.cc:654] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:40.062249 28194 log.cc:824] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:40.062551 28195 tablet_bootstrap.cc:654] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:40.063637 28195 log.cc:824] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:40.065362 28194 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: No bootstrap required, opened a new log
I20260925 11:11:40.065654 28194 ts_tablet_manager.cc:1397] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 11:11:40.066658 28195 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: No bootstrap required, opened a new log
I20260925 11:11:40.066912 28195 ts_tablet_manager.cc:1397] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260925 11:11:40.069097 28198 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Bootstrap starting.
I20260925 11:11:40.072932 28198 tablet_bootstrap.cc:654] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:40.073911 28198 log.cc:824] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:40.076773 28198 tablet_bootstrap.cc:492] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: No bootstrap required, opened a new log
I20260925 11:11:40.076678 28194 raft_consensus.cc:348] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.077045 28198 ts_tablet_manager.cc:1397] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 11:11:40.077132 28194 raft_consensus.cc:374] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:40.077234 28194 raft_consensus.cc:729] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f69ee4bc1824f45a762b762edb52558, State: Initialized, Role: FOLLOWER
I20260925 11:11:40.077210 28195 raft_consensus.cc:348] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.077622 28195 raft_consensus.cc:374] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:40.077728 28195 raft_consensus.cc:729] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4f25be019448089bf91fb9df83ad99, State: Initialized, Role: FOLLOWER
I20260925 11:11:40.077981 28194 consensus_queue.cc:260] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.078274 28195 consensus_queue.cc:260] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.080086 28194 ts_tablet_manager.cc:1428] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 11:11:40.080981 28046 heartbeater.cc:503] Master 127.8.181.62:38203 was elected leader, sending a full tablet report...
I20260925 11:11:40.081637 28195 ts_tablet_manager.cc:1428] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260925 11:11:40.087073 28198 raft_consensus.cc:348] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.087440 28198 raft_consensus.cc:374] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:40.087553 28198 raft_consensus.cc:729] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607c2e4fc1a14faa933d2a078e17ef4e, State: Initialized, Role: FOLLOWER
I20260925 11:11:40.088157 28198 consensus_queue.cc:260] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.089131 28176 heartbeater.cc:503] Master 127.8.181.62:38203 was elected leader, sending a full tablet report...
I20260925 11:11:40.089598 28198 ts_tablet_manager.cc:1428] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 11:11:40.091539 28201 raft_consensus.cc:482] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:40.091799 28201 raft_consensus.cc:504] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.093477 28201 leader_election.cc:288] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f69ee4bc1824f45a762b762edb52558 (127.8.181.1:42725), 607c2e4fc1a14faa933d2a078e17ef4e (127.8.181.3:45319)
I20260925 11:11:40.100000 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.093821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:48637 (local address 127.8.181.1:42725)
0925 11:11:40.094013 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.094021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.094394 (+ 373us) server_negotiation.cc:407] Connection header received
0925 11:11:40.094577 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.094582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.094699 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.094813 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:40.095410 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.096281 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.098123 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.099086 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.099229 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.099327 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.099641 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.099763 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:11:40.100065 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.093641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.8.181.2:48637)
0925 11:11:40.094283 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.094301 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:40.094416 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:40.094831 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:40.094834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:40.094855 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:40.095271 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.095285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.096389 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.096397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.097908 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.097919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.099271 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.099277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.099446 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.099509 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 11:11:40.099615 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:11:40.099745 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":314,"threads_started":1}
I20260925 11:11:40.101255 27870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488ba0c4b011452ea358ffa1960fad75" candidate_uuid: "bc4f25be019448089bf91fb9df83ad99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f69ee4bc1824f45a762b762edb52558" is_pre_election: true
I20260925 11:11:40.101847 27870 raft_consensus.cc:2454] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4f25be019448089bf91fb9df83ad99 in term 0.
I20260925 11:11:40.102540 27936 leader_election.cc:302] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f69ee4bc1824f45a762b762edb52558, bc4f25be019448089bf91fb9df83ad99; no voters:
I20260925 11:11:40.102960 28201 raft_consensus.cc:2781] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:40.103122 28201 raft_consensus.cc:482] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:40.103245 28201 raft_consensus.cc:3037] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:40.106076 28203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.093765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:45319 (local address 127.8.181.2:38075)
0925 11:11:40.100368 (+ 6603us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.100383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:40.100501 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:40.100967 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:40.100971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:40.101001 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:40.101182 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.101196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.102607 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.102611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.104031 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:40.104043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.105571 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.105575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:40.105698 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.105721 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 11:11:40.105789 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 11:11:40.105876 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6515,"thread_start_us":7355,"threads_started":1}
I20260925 11:11:40.106231 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.094106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:38075 (local address 127.8.181.3:45319)
0925 11:11:40.094258 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.094265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.100457 (+ 6192us) server_negotiation.cc:407] Connection header received
0925 11:11:40.100629 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.100635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.100719 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.100804 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:40.101371 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.102390 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.104182 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.105397 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.105524 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.105621 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.105830 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.105958 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 11:11:40.106252 28201 raft_consensus.cc:504] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.107106 28131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488ba0c4b011452ea358ffa1960fad75" candidate_uuid: "bc4f25be019448089bf91fb9df83ad99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" is_pre_election: true
I20260925 11:11:40.107352 28201 leader_election.cc:288] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [CANDIDATE]: Term 1 election: Requested vote from peers 6f69ee4bc1824f45a762b762edb52558 (127.8.181.1:42725), 607c2e4fc1a14faa933d2a078e17ef4e (127.8.181.3:45319)
I20260925 11:11:40.107637 28131 raft_consensus.cc:2454] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc4f25be019448089bf91fb9df83ad99 in term 0.
I20260925 11:11:40.107719 27870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488ba0c4b011452ea358ffa1960fad75" candidate_uuid: "bc4f25be019448089bf91fb9df83ad99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f69ee4bc1824f45a762b762edb52558"
I20260925 11:11:40.107944 27870 raft_consensus.cc:3037] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:40.108080 28130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488ba0c4b011452ea358ffa1960fad75" candidate_uuid: "bc4f25be019448089bf91fb9df83ad99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607c2e4fc1a14faa933d2a078e17ef4e"
I20260925 11:11:40.108333 28130 raft_consensus.cc:3037] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:40.110242 27870 raft_consensus.cc:2454] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4f25be019448089bf91fb9df83ad99 in term 1.
I20260925 11:11:40.110562 28130 raft_consensus.cc:2454] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc4f25be019448089bf91fb9df83ad99 in term 1.
I20260925 11:11:40.110620 27936 leader_election.cc:302] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f69ee4bc1824f45a762b762edb52558, bc4f25be019448089bf91fb9df83ad99; no voters:
I20260925 11:11:40.110925 28201 raft_consensus.cc:2781] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:40.111400 28201 raft_consensus.cc:686] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 1 LEADER]: Becoming Leader. State: Replica: bc4f25be019448089bf91fb9df83ad99, State: Running, Role: LEADER
I20260925 11:11:40.111934 28201 consensus_queue.cc:237] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } }
I20260925 11:11:40.116436 27728 catalog_manager.cc:5495] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc4f25be019448089bf91fb9df83ad99 (127.8.181.2). New cstate: current_term: 1 leader_uuid: "bc4f25be019448089bf91fb9df83ad99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc4f25be019448089bf91fb9df83ad99" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 } health_report { overall_health: UNKNOWN } } }
I20260925 11:11:40.143472 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.138309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.8.181.2:43987)
0925 11:11:40.138545 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.138553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.138572 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:11:40.138662 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.138665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.138749 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.138899 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:40.139270 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.139950 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.141259 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.141657 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.141790 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.142366 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:40.142909 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:40.143103 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.143185 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.143226 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 11:11:40.151309 28130 raft_consensus.cc:1261] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 1 FOLLOWER]: Refusing update from remote peer bc4f25be019448089bf91fb9df83ad99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:40.151309 27870 raft_consensus.cc:1261] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558 [term 1 FOLLOWER]: Refusing update from remote peer bc4f25be019448089bf91fb9df83ad99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:40.152235 28205 consensus_queue.cc:1035] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607c2e4fc1a14faa933d2a078e17ef4e" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:40.152616 28201 consensus_queue.cc:1035] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f69ee4bc1824f45a762b762edb52558" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 42725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:40.167798 28209 mvcc.cc:205] Tried to move back new op lower bound from 7333210931810021376 to 7333210931662913536. Current Snapshot: MvccSnapshot[applied={T|T < 7333210931810021376}]
I20260925 11:11:40.179643 27728 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:40.179991 27728 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:40.183566 27728 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a647ded6b0ba4f57a4aff012ccac678e: Sending DeleteTablet for 3 replicas of tablet 488ba0c4b011452ea358ffa1960fad75
I20260925 11:11:40.184808 28111 tablet_service.cc:1514] Processing DeleteTablet for tablet 488ba0c4b011452ea358ffa1960fad75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:40 UTC) from {username='slave'} at 127.0.0.1:58236
I20260925 11:11:40.184814 27981 tablet_service.cc:1514] Processing DeleteTablet for tablet 488ba0c4b011452ea358ffa1960fad75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:40 UTC) from {username='slave'} at 127.0.0.1:33728
I20260925 11:11:40.184835 27850 tablet_service.cc:1514] Processing DeleteTablet for tablet 488ba0c4b011452ea358ffa1960fad75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:40 UTC) from {username='slave'} at 127.0.0.1:47054
I20260925 11:11:40.187594 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27788
I20260925 11:11:40.197615 28219 tablet_replica.cc:330] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: stopping tablet replica
I20260925 11:11:40.198225 28219 raft_consensus.cc:2229] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:40.198956 28218 tablet_replica.cc:330] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: stopping tablet replica
I20260925 11:11:40.199625 28218 raft_consensus.cc:2229] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:40.200312 28218 raft_consensus.cc:2258] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:40.201454 28219 raft_consensus.cc:2258] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:40.204097 28219 ts_tablet_manager.cc:1905] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:40.213104 28218 ts_tablet_manager.cc:1905] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:40.220155 28218 ts_tablet_manager.cc:1918] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:40.220359 28218 log.cc:1196] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/488ba0c4b011452ea358ffa1960fad75
I20260925 11:11:40.220908 28218 ts_tablet_manager.cc:1939] T 488ba0c4b011452ea358ffa1960fad75 P 607c2e4fc1a14faa933d2a078e17ef4e: Deleting consensus metadata
I20260925 11:11:40.223009 27714 catalog_manager.cc:4841] TS 607c2e4fc1a14faa933d2a078e17ef4e (127.8.181.3:45319): tablet 488ba0c4b011452ea358ffa1960fad75 (table test-table [id=67ddc8a371d745ecb1bdfcca0b6035fe]) successfully deleted
I20260925 11:11:40.223798 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.189157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.8.181.1:42725)
0925 11:11:40.189387 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.189409 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.212711 (+ 23302us) server_negotiation.cc:407] Connection header received
0925 11:11:40.214873 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.214880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.215005 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.215240 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:40.215989 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.217161 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.219178 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.219868 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.219993 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.221912 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:40.221924 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:40.221942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:40.222768 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:40.222831 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:40.222866 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:40.222946 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.223280 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.223407 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:11:40.225239 27910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:40.226825 28219 ts_tablet_manager.cc:1918] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:40.227033 28219 log.cc:1196] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/488ba0c4b011452ea358ffa1960fad75
I20260925 11:11:40.227659 28219 ts_tablet_manager.cc:1939] T 488ba0c4b011452ea358ffa1960fad75 P bc4f25be019448089bf91fb9df83ad99: Deleting consensus metadata
I20260925 11:11:40.229653 27714 catalog_manager.cc:4841] TS bc4f25be019448089bf91fb9df83ad99 (127.8.181.2:43987): tablet 488ba0c4b011452ea358ffa1960fad75 (table test-table [id=67ddc8a371d745ecb1bdfcca0b6035fe]) successfully deleted
I20260925 11:11:40.316772 28220 tablet_replica.cc:330] T 488ba0c4b011452ea358ffa1960fad75 P 6f69ee4bc1824f45a762b762edb52558: stopping tablet replica
I20260925 11:11:40.317116 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27788
W20260925 11:11:40.334757 27715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.181.1:42725 (error 108)
I20260925 11:11:40.334952 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27919
I20260925 11:11:40.337675 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.337208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47080)
0925 11:11:40.337401 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.337502 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 11:11:40.337946 27715 catalog_manager.cc:4584] TS 6f69ee4bc1824f45a762b762edb52558 (127.8.181.1:42725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 488ba0c4b011452ea358ffa1960fad75: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
I20260925 11:11:40.343534 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.336012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.8.181.2:43987)
0925 11:11:40.336218 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.336225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.336506 (+ 281us) server_negotiation.cc:407] Connection header received
0925 11:11:40.336584 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.336589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.336701 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.336826 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:40.337276 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.338659 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.340003 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.340675 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.340809 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.341839 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:40.341850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:40.341869 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:40.342715 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:40.342773 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:40.342804 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:40.342928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.343073 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.343164 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:11:40.344266 28041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:40.391558 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.391113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47086)
0925 11:11:40.391318 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.391411 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 11:11:40.392679 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.392261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47088)
0925 11:11:40.392414 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.392528 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 11:11:40.453316 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.452842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47092)
0925 11:11:40.453073 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.453167 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 11:11:40.454507 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.454043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47102)
0925 11:11:40.454197 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.454283 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 11:11:40.455752 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27919
I20260925 11:11:40.474436 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28049
I20260925 11:11:40.486694 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.475320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.8.181.3:45319)
0925 11:11:40.475513 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:40.475519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:40.475541 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:11:40.475669 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:40.475673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:40.475759 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:40.475846 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:40.476263 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.476976 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.478256 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:40.478822 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:40.478942 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:40.485174 (+ 6232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:40.485185 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:40.485200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:40.485790 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:40.485845 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:40.485869 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:40.486076 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:40.486199 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 11:11:40.486303 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 11:11:40.487361 28171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:40.535470 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.534982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47108)
0925 11:11:40.535201 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.535301 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 11:11:40.536610 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.536158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47118)
0925 11:11:40.536358 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.536451 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 11:11:40.582314 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28049
I20260925 11:11:40.599360 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 27698
I20260925 11:11:40.600664 27759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:40.710882 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:40.710229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:42725 (local address 127.0.0.1:47124)
0925 11:11:40.710559 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:40.710681 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.1:42725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 11:11:40.710873 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 27698
2026-09-25T11:11:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2784 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:40 dist-test-slave-6042 krb5kdc[28229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:40 dist-test-slave-6042 krb5kdc[28229](info): set up 2 sockets
Sep 25 11:11:40 dist-test-slave-6042 krb5kdc[28229](info): commencing operation
krb5kdc: starting...
W20260925 11:11:41.895583 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:42 dist-test-slave-6042 krb5kdc[28229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:11:42.166144 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35297
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:42.499125 28242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:42.499529 28242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:42.499693 28242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:42.507112 28242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:42.507275 28242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:42.507381 28242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:42.507477 28242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:42.507550 28242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:42.518023 28242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:35297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:42.519474 28242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:42.520807 28242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:42.535506 28247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:42.535475 28248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:42.535938 28250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:42.536887 28242 server_base.cc:1034] running on GCE node
I20260925 11:11:42.537716 28242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:42.539304 28242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:42.540570 28242 hybrid_clock.cc:648] HybridClock initialized: now 1790334702540513 us; error 57 us; skew 500 ppm
I20260925 11:11:42.541081 28242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:42 dist-test-slave-6042 krb5kdc[28229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:42.544617 28242 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:11:42.546281 28242 webserver.cc:460] Webserver started at http://127.8.181.62:34819/ using document root <none> and password file <none>
I20260925 11:11:42.547070 28242 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:42.547222 28242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:42.547545 28242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:42.550748 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "71f28e01bc8f4da7838e6e3c2e906946"
format_stamp: "Formatted at 2026-09-25 11:11:42 on dist-test-slave-6042"
server_key: "75a39bda186686c34ff954014c1f3e79"
server_key_iv: "941a1f01a6f3c1b7203a4892c69d8390"
server_key_version: "encryptionkey@0"
I20260925 11:11:42.551528 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "71f28e01bc8f4da7838e6e3c2e906946"
format_stamp: "Formatted at 2026-09-25 11:11:42 on dist-test-slave-6042"
server_key: "75a39bda186686c34ff954014c1f3e79"
server_key_iv: "941a1f01a6f3c1b7203a4892c69d8390"
server_key_version: "encryptionkey@0"
I20260925 11:11:42.556850 28242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 11:11:42.560360 28257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:42.561666 28242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:42.561854 28242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "71f28e01bc8f4da7838e6e3c2e906946"
format_stamp: "Formatted at 2026-09-25 11:11:42 on dist-test-slave-6042"
server_key: "75a39bda186686c34ff954014c1f3e79"
server_key_iv: "941a1f01a6f3c1b7203a4892c69d8390"
server_key_version: "encryptionkey@0"
I20260925 11:11:42.562062 28242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:42.579152 28242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:42.583830 28242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:42.584141 28242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:42.603502 28242 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:35297
I20260925 11:11:42.603541 28308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:35297 every 8 connection(s)
I20260925 11:11:42.604887 28242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:42.608695 28309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:42.611555 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28242
I20260925 11:11:42.611770 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:42.612336 8916 external_mini_cluster.cc:1428] Setting key 5f89b1f0324cace965d37e2b66351453
I20260925 11:11:42.621423 28309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: Bootstrap starting.
I20260925 11:11:42.625895 28309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:42.627405 28309 log.cc:824] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: Log is configured to *not* fsync() on all Append() calls
Sep 25 11:11:42 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:42.631145 28309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: No bootstrap required, opened a new log
I20260925 11:11:42.637897 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:42.614248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.8.181.62:35297)
0925 11:11:42.615130 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:42.615141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:42.615182 (+ 41us) server_negotiation.cc:407] Connection header received
0925 11:11:42.616814 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:42.616838 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:42.617273 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:42.617614 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:42.619433 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:42.620723 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:42.622587 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:42.623323 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:42.623528 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:42.629087 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:42.629098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:42.629118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:42.629168 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:42.633290 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:42.634231 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:42.634241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:42.634250 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:42.634488 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:42.634959 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:42.634965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:42.634968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:42.635543 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:42.635780 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:42.636300 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 11:11:42.636743 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":468,"threads_started":1}
I20260925 11:11:42.641048 28309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } }
I20260925 11:11:42.641495 28309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:42.641640 28309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f28e01bc8f4da7838e6e3c2e906946, State: Initialized, Role: FOLLOWER
I20260925 11:11:42.642302 28309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } }
I20260925 11:11:42.642594 28309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:42.642760 28309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:42.642943 28309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:42.645231 28309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } }
I20260925 11:11:42.645783 28309 leader_election.cc:302] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71f28e01bc8f4da7838e6e3c2e906946; no voters:
I20260925 11:11:42.646409 28309 leader_election.cc:288] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:42.646620 28314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:42.647656 28314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [term 1 LEADER]: Becoming Leader. State: Replica: 71f28e01bc8f4da7838e6e3c2e906946, State: Running, Role: LEADER
I20260925 11:11:42.648293 28314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } }
I20260925 11:11:42.649053 28309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:42.651161 28316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f28e01bc8f4da7838e6e3c2e906946. Latest consensus state: current_term: 1 leader_uuid: "71f28e01bc8f4da7838e6e3c2e906946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } } }
I20260925 11:11:42.651625 28316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:42.651175 28315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71f28e01bc8f4da7838e6e3c2e906946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f28e01bc8f4da7838e6e3c2e906946" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 35297 } } }
I20260925 11:11:42.654043 28315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946 [sys.catalog]: This master's current role is: LEADER
W20260925 11:11:42.661903 28330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:11:42.662045 28330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:11:42.662195 28319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:42.664533 28319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:42.673869 28319 catalog_manager.cc:1347] Generated new cluster ID: a35014e7fdbe40f2baf773a00daccb39
I20260925 11:11:42.674005 28319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:42.700824 28319 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:42.702284 28319 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:42.715318 28319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: Generated new TSK 0
I20260925 11:11:42.716428 28319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.181.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
Entry for principal kudu/127.8.181.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab.
I20260925 11:11:42.794564 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:43.149897 28335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:43.150274 28335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:43.150424 28335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:43.158614 28335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:43.158737 28335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:43.158946 28335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:43.169474 28335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.1.keytab
--principal=kudu/127.8.181.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:43.171044 28335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:43.172487 28335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:43.186750 28343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:43.186843 28340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:43.186764 28341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:43.187683 28335 server_base.cc:1034] running on GCE node
I20260925 11:11:43.188364 28335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:43.189392 28335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:43.190658 28335 hybrid_clock.cc:648] HybridClock initialized: now 1790334703190582 us; error 64 us; skew 500 ppm
I20260925 11:11:43.191036 28335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:43 dist-test-slave-6042 krb5kdc[28229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334703, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:43.194348 28335 init.cc:377] Logged in from keytab as kudu/127.8.181.1@KRBTEST.COM (short username kudu)
I20260925 11:11:43.195982 28335 webserver.cc:460] Webserver started at http://127.8.181.1:39621/ using document root <none> and password file <none>
I20260925 11:11:43.196705 28335 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:43.196846 28335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:43.197157 28335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:43.200207 28335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "67965fc339d547ae9e8172b4249ff2c9"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "0be64b98b3fc69b0ac4e123e2d6b2c9a"
server_key_iv: "0c2aff3c43fb77807e721d2e82d9b012"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.201020 28335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "67965fc339d547ae9e8172b4249ff2c9"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "0be64b98b3fc69b0ac4e123e2d6b2c9a"
server_key_iv: "0c2aff3c43fb77807e721d2e82d9b012"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.206441 28335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 11:11:43.210595 28350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.212199 28335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 11:11:43.212422 28335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "67965fc339d547ae9e8172b4249ff2c9"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "0be64b98b3fc69b0ac4e123e2d6b2c9a"
server_key_iv: "0c2aff3c43fb77807e721d2e82d9b012"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.212672 28335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:43.232867 28335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:43.237383 28335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:43.237701 28335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:43.238794 28335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:43.240512 28335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:43.240620 28335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.240784 28335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:43.240868 28335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.276261 28335 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:41751
I20260925 11:11:43.276310 28462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:41751 every 8 connection(s)
I20260925 11:11:43.277602 28335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:43.282910 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28335
I20260925 11:11:43.283134 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:43.283568 8916 external_mini_cluster.cc:1428] Setting key 21cc61b299d6439a86643814074106b0
Sep 25 11:11:43 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334703, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.1@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:43.299690 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:43.280233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:38385 (local address 127.8.181.62:35297)
0925 11:11:43.280679 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:43.280684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:43.281375 (+ 691us) server_negotiation.cc:407] Connection header received
0925 11:11:43.283416 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:43.283422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:43.283515 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:43.283629 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:43.285543 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.286649 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.288306 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.289006 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.289156 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:43.293027 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:43.293037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:43.293045 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:43.293092 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:43.296177 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:43.297198 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:43.297202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:43.297205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:43.297323 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:43.297828 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:43.297831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:43.297833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:43.298093 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:43.298205 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:43.299233 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0925 11:11:43.299420 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":244,"threads_started":1}
I20260925 11:11:43.301307 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:43.280528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35297 (local address 127.8.181.1:38385)
0925 11:11:43.281212 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:43.281249 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:43.283120 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:43.283877 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:43.283885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:43.284556 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:43.285362 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:43.285392 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.286809 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.286814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:43.288138 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:43.288151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.289239 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.289245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:43.289456 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:43.290197 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:43.290225 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:43.292765 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:43.296345 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:43.296361 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:43.296383 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:43.297072 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:43.297438 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:43.297446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:43.297456 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:43.297718 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:43.298226 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:43.298238 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:43.298801 (+ 563us) client_negotiation.cc:769] Sending connection context
0925 11:11:43.299162 (+ 361us) client_negotiation.cc:240] Negotiation successful
0925 11:11:43.299793 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20260925 11:11:43.303069 28463 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35297
I20260925 11:11:43.303406 28463 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:43.304184 28463 heartbeater.cc:511] Master 127.8.181.62:35297 requested a full tablet report, sending...
I20260925 11:11:43.306453 28274 ts_manager.cc:194] Registered new tserver with Master: 67965fc339d547ae9e8172b4249ff2c9 (127.8.181.1:41751)
I20260925 11:11:43.307965 28274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.1@KRBTEST.COM'} at 127.8.181.1:38385
WARNING: no policy specified for kudu/127.8.181.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
Entry for principal kudu/127.8.181.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab.
I20260925 11:11:43.346299 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:43.693115 28469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:43.693480 28469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:43.693645 28469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:43.701185 28469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:43.701377 28469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:43.701566 28469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:43.711936 28469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.2.keytab
--principal=kudu/127.8.181.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:43.713361 28469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:43.714788 28469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:43.728837 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:43.728863 28474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:43.728838 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:43.729750 28469 server_base.cc:1034] running on GCE node
I20260925 11:11:43.730602 28469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:43.731698 28469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:43.732963 28469 hybrid_clock.cc:648] HybridClock initialized: now 1790334703732887 us; error 67 us; skew 500 ppm
I20260925 11:11:43.733343 28469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:43 dist-test-slave-6042 krb5kdc[28229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334703, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:43.736662 28469 init.cc:377] Logged in from keytab as kudu/127.8.181.2@KRBTEST.COM (short username kudu)
I20260925 11:11:43.738272 28469 webserver.cc:460] Webserver started at http://127.8.181.2:42701/ using document root <none> and password file <none>
I20260925 11:11:43.739081 28469 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:43.739238 28469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:43.739554 28469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:43.742689 28469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "fd42a5860eb443efbd7ceda935a4d6aa"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "821cc86b4655886d7052ffb1c5201225"
server_key_iv: "b41f34019828ee37b8a20dc5eac35c19"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.743546 28469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "fd42a5860eb443efbd7ceda935a4d6aa"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "821cc86b4655886d7052ffb1c5201225"
server_key_iv: "b41f34019828ee37b8a20dc5eac35c19"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.750448 28469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 11:11:43.754541 28484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.756256 28469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:43.756448 28469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "fd42a5860eb443efbd7ceda935a4d6aa"
format_stamp: "Formatted at 2026-09-25 11:11:43 on dist-test-slave-6042"
server_key: "821cc86b4655886d7052ffb1c5201225"
server_key_iv: "b41f34019828ee37b8a20dc5eac35c19"
server_key_version: "encryptionkey@0"
I20260925 11:11:43.756703 28469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:43.787506 28469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:43.792039 28469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:43.792373 28469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:43.793442 28469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:43.794907 28469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:43.795009 28469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.795164 28469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:43.795243 28469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:43.830291 28469 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:39453
I20260925 11:11:43.830340 28596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:39453 every 8 connection(s)
I20260925 11:11:43.831736 28469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:43.834175 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28469
I20260925 11:11:43.834369 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:43.834951 8916 external_mini_cluster.cc:1428] Setting key a836e2416c7fa2475a78d59bef0a380f
Sep 25 11:11:43 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334703, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.2@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:43.856011 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:43.834570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:60835 (local address 127.8.181.62:35297)
0925 11:11:43.835173 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:43.835177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:43.836210 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 11:11:43.838420 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:43.838438 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:43.838566 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:43.838691 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:43.840408 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.841588 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.843394 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.844151 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.844299 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:43.848385 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:43.848396 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:43.848405 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:43.848455 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:43.852271 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:43.853198 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:43.853207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:43.853214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:43.853369 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:43.853817 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:43.853825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:43.853831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:43.854251 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:43.854403 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:43.855260 (+ 857us) server_negotiation.cc:299] Negotiation successful
0925 11:11:43.855526 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":263,"threads_started":1}
I20260925 11:11:43.856813 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:43.835147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35297 (local address 127.8.181.2:60835)
0925 11:11:43.836025 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:43.836090 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:43.838116 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:43.838988 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:43.838998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:43.839598 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:43.840208 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:43.840226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.841762 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.841765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:43.843223 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:43.843237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:43.844397 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:43.844402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:43.844621 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:43.845353 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:43.845376 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:43.848111 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:43.852502 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:43.852513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:43.852539 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:43.853037 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:43.853482 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:43.853485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:43.853488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:43.853683 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:43.854384 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:43.854391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:43.854835 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 11:11:43.855091 (+ 256us) client_negotiation.cc:240] Negotiation successful
0925 11:11:43.855585 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":357,"threads_started":1}
I20260925 11:11:43.858392 28597 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35297
I20260925 11:11:43.858731 28597 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:43.859483 28597 heartbeater.cc:511] Master 127.8.181.62:35297 requested a full tablet report, sending...
I20260925 11:11:43.861178 28274 ts_manager.cc:194] Registered new tserver with Master: fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453)
I20260925 11:11:43.862182 28274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.2@KRBTEST.COM'} at 127.8.181.2:60835
WARNING: no policy specified for kudu/127.8.181.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
Entry for principal kudu/127.8.181.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab.
I20260925 11:11:43.898689 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:44.244592 28603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:44.244942 28603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:44.245118 28603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:44.252746 28603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:44.252923 28603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:44.253091 28603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:44.263518 28603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.3.keytab
--principal=kudu/127.8.181.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:35297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:44.264923 28603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:44.266448 28603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:44.280370 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:44.280416 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:44.280447 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:44.282078 28603 server_base.cc:1034] running on GCE node
I20260925 11:11:44.282800 28603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:44.283880 28603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:44.285131 28603 hybrid_clock.cc:648] HybridClock initialized: now 1790334704285058 us; error 61 us; skew 500 ppm
I20260925 11:11:44.285527 28603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:44 dist-test-slave-6042 krb5kdc[28229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334704, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:44.289605 28603 init.cc:377] Logged in from keytab as kudu/127.8.181.3@KRBTEST.COM (short username kudu)
I20260925 11:11:44.291503 28603 webserver.cc:460] Webserver started at http://127.8.181.3:35175/ using document root <none> and password file <none>
I20260925 11:11:44.292312 28603 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:44.292459 28603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:44.292816 28603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:44.296123 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "15f1b197211a453f863126cafbfee5c3"
format_stamp: "Formatted at 2026-09-25 11:11:44 on dist-test-slave-6042"
server_key: "a801118394fb92054fedf5f6e274e575"
server_key_iv: "09cadf749182af2813888d2087cf1a71"
server_key_version: "encryptionkey@0"
I20260925 11:11:44.296948 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "15f1b197211a453f863126cafbfee5c3"
format_stamp: "Formatted at 2026-09-25 11:11:44 on dist-test-slave-6042"
server_key: "a801118394fb92054fedf5f6e274e575"
server_key_iv: "09cadf749182af2813888d2087cf1a71"
server_key_version: "encryptionkey@0"
I20260925 11:11:44.302673 28603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 11:11:44.306565 28618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:44.307968 28603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:11:44.308162 28603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "15f1b197211a453f863126cafbfee5c3"
format_stamp: "Formatted at 2026-09-25 11:11:44 on dist-test-slave-6042"
server_key: "a801118394fb92054fedf5f6e274e575"
server_key_iv: "09cadf749182af2813888d2087cf1a71"
server_key_version: "encryptionkey@0"
I20260925 11:11:44.308386 28603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:44.311141 28463 heartbeater.cc:503] Master 127.8.181.62:35297 was elected leader, sending a full tablet report...
I20260925 11:11:44.333241 28603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:44.337873 28603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:44.338228 28603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:44.339334 28603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:44.340939 28603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:44.341022 28603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:44.341122 28603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:44.341167 28603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:44.375468 28603 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:37985
I20260925 11:11:44.375532 28730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:37985 every 8 connection(s)
I20260925 11:11:44.376878 28603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:44.385788 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28603
I20260925 11:11:44.385989 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:44.386544 8916 external_mini_cluster.cc:1428] Setting key 822b3ba9bed1b82f65c7dfdcc85ecf5f
Sep 25 11:11:44 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334704, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.3@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:11:44.397629 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.379700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:54237 (local address 127.8.181.62:35297)
0925 11:11:44.380189 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.380194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.381302 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 11:11:44.383281 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.383286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.383357 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.383459 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:44.385029 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.385844 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.387705 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.388125 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.388260 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.391798 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:44.391803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:44.391806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:44.391841 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:44.394529 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.395274 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.395285 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.395288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.395400 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.395823 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.395828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.395832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.396107 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:44.396280 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.396979 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.397209 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260925 11:11:44.398640 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.380245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:35297 (local address 127.8.181.3:54237)
0925 11:11:44.381111 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.381169 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.383012 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.383708 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.383716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.384228 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:44.384860 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.384877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.386011 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.386015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.387560 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.387573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.388352 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.388359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.388607 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.389282 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:11:44.389313 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:44.391566 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:44.394675 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:44.394684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:44.394702 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:44.395145 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:44.395508 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:11:44.395513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:11:44.395516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:11:44.395707 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:11:44.396246 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:44.396255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:44.396609 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.396874 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.397446 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":375,"threads_started":1}
I20260925 11:11:44.400261 28731 heartbeater.cc:347] Connected to a master server at 127.8.181.62:35297
I20260925 11:11:44.400624 28731 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:44.401543 28731 heartbeater.cc:511] Master 127.8.181.62:35297 requested a full tablet report, sending...
I20260925 11:11:44.403090 28274 ts_manager.cc:194] Registered new tserver with Master: 15f1b197211a453f863126cafbfee5c3 (127.8.181.3:37985)
I20260925 11:11:44.403913 28274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.181.3@KRBTEST.COM'} at 127.8.181.3:54237
I20260925 11:11:44.414572 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:44.431381 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.418956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.8.181.62:35297)
0925 11:11:44.419207 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.419217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.419682 (+ 465us) server_negotiation.cc:407] Connection header received
0925 11:11:44.419815 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.419819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.419886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.420021 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:44.421405 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.422211 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.423524 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.423918 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.424017 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.426344 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:44.426349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:44.426353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:44.426380 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:44.428788 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.429437 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.429442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.429444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.429568 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.429867 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.429871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.429873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.430186 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:44.430364 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.430851 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.431029 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:11:44.443393 28274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:44.450582 28274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:44.492287 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.463887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.8.181.1:41751)
0925 11:11:44.482173 (+ 18286us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.482187 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.482230 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:11:44.482377 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.482390 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.482802 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.483151 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.484577 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.486324 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.489903 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.490992 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.491131 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.491401 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.491752 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.491930 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18061,"thread_start_us":17876,"threads_started":1}
I20260925 11:11:44.492388 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.463650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41751 (local address 127.0.0.1:58434)
0925 11:11:44.477896 (+ 14246us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.477976 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.478273 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.483554 (+ 5281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.483567 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.483639 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.484246 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.484265 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.486564 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.486570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.489731 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.489755 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.491228 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.491235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.491466 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.491521 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.491671 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.491934 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13830,"mutex_wait_us":3,"thread_start_us":13631,"threads_started":1}
I20260925 11:11:44.495455 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.464918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.8.181.2:39453)
0925 11:11:44.486437 (+ 21519us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.486444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.486523 (+ 79us) server_negotiation.cc:407] Connection header received
0925 11:11:44.486640 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.486649 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.486869 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.487065 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.487758 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.489127 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.493219 (+ 4092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.494368 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.494549 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.494750 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.494974 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.495126 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21374,"thread_start_us":21265,"threads_started":1}
I20260925 11:11:44.495488 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.464193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:39453 (local address 127.0.0.1:47690)
0925 11:11:44.483933 (+ 19740us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.483963 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.484140 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.487082 (+ 2942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.487103 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.487128 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.487601 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.487610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.489309 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.489315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.493009 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.493023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.494597 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.494604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.494771 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.494813 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.494922 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.495062 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19530,"mutex_wait_us":1504,"thread_start_us":20074,"threads_started":1}
I20260925 11:11:44.496249 28398 tablet_service.cc:1467] Processing CreateTablet for tablet 855f6b73e0a94a12ae8d271233f16031 (DEFAULT_TABLE table=test-table [id=20878f6fde694f23b2a2f926fa56c52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:44.497140 28747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.464426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:37985 (local address 127.0.0.1:58460)
0925 11:11:44.485703 (+ 21277us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.485728 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.485904 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.486720 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.486724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.486758 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.487231 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.487250 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.489197 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.489202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.491000 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.491014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.493593 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.493600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.496495 (+ 2895us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.496520 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.496636 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.496770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21031,"mutex_wait_us":3,"thread_start_us":20854,"threads_started":1}
I20260925 11:11:44.497208 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.465445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58460 (local address 127.8.181.3:37985)
0925 11:11:44.485057 (+ 19612us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.485072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.486048 (+ 976us) server_negotiation.cc:407] Connection header received
0925 11:11:44.486167 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.486174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.486502 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.486730 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.487476 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.489022 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.491193 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.493039 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.493216 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.493452 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.496695 (+ 3243us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.496877 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19421,"thread_start_us":19256,"threads_started":1}
I20260925 11:11:44.497838 28398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855f6b73e0a94a12ae8d271233f16031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:44.499128 28532 tablet_service.cc:1467] Processing CreateTablet for tablet 855f6b73e0a94a12ae8d271233f16031 (DEFAULT_TABLE table=test-table [id=20878f6fde694f23b2a2f926fa56c52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:44.500578 28532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855f6b73e0a94a12ae8d271233f16031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:44.500550 28666 tablet_service.cc:1467] Processing CreateTablet for tablet 855f6b73e0a94a12ae8d271233f16031 (DEFAULT_TABLE table=test-table [id=20878f6fde694f23b2a2f926fa56c52e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:44.501735 28666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 855f6b73e0a94a12ae8d271233f16031. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:44.519145 28751 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Bootstrap starting.
I20260925 11:11:44.524480 28752 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Bootstrap starting.
I20260925 11:11:44.524941 28750 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Bootstrap starting.
I20260925 11:11:44.526777 28751 tablet_bootstrap.cc:654] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:44.529066 28752 tablet_bootstrap.cc:654] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:44.529136 28750 tablet_bootstrap.cc:654] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:44.529026 28751 log.cc:824] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:44.530494 28752 log.cc:824] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:44.530503 28750 log.cc:824] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:44.532742 28751 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: No bootstrap required, opened a new log
I20260925 11:11:44.533052 28751 ts_tablet_manager.cc:1397] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260925 11:11:44.533540 28750 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: No bootstrap required, opened a new log
I20260925 11:11:44.533900 28750 ts_tablet_manager.cc:1397] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 11:11:44.534639 28752 tablet_bootstrap.cc:492] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: No bootstrap required, opened a new log
I20260925 11:11:44.534951 28752 ts_tablet_manager.cc:1397] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260925 11:11:44.542972 28751 raft_consensus.cc:348] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.543406 28751 raft_consensus.cc:374] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:44.543529 28751 raft_consensus.cc:729] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd42a5860eb443efbd7ceda935a4d6aa, State: Initialized, Role: FOLLOWER
I20260925 11:11:44.544165 28751 consensus_queue.cc:260] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.545357 28597 heartbeater.cc:503] Master 127.8.181.62:35297 was elected leader, sending a full tablet report...
I20260925 11:11:44.545599 28752 raft_consensus.cc:348] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.545908 28751 ts_tablet_manager.cc:1428] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260925 11:11:44.546037 28752 raft_consensus.cc:374] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:44.546136 28752 raft_consensus.cc:729] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15f1b197211a453f863126cafbfee5c3, State: Initialized, Role: FOLLOWER
I20260925 11:11:44.545027 28750 raft_consensus.cc:348] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.546893 28750 raft_consensus.cc:374] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:44.546756 28752 consensus_queue.cc:260] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.547011 28750 raft_consensus.cc:729] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67965fc339d547ae9e8172b4249ff2c9, State: Initialized, Role: FOLLOWER
I20260925 11:11:44.547569 28750 consensus_queue.cc:260] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.548133 28731 heartbeater.cc:503] Master 127.8.181.62:35297 was elected leader, sending a full tablet report...
I20260925 11:11:44.548794 28752 ts_tablet_manager.cc:1428] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260925 11:11:44.550619 28750 ts_tablet_manager.cc:1428] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260925 11:11:44.583912 28598 tablet.cc:2369] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 11:11:44.628958 28732 tablet.cc:2369] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:44.666584 28757 raft_consensus.cc:482] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:44.666899 28757 raft_consensus.cc:504] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.668612 28757 leader_election.cc:288] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67965fc339d547ae9e8172b4249ff2c9 (127.8.181.1:41751), fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453)
I20260925 11:11:44.671615 28758 raft_consensus.cc:482] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:44.671959 28758 raft_consensus.cc:504] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.674067 28758 leader_election.cc:288] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453), 15f1b197211a453f863126cafbfee5c3 (127.8.181.3:37985)
I20260925 11:11:44.676191 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.669328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:36645 (local address 127.8.181.2:39453)
0925 11:11:44.669504 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.669510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.669534 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:11:44.669634 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.669638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.669697 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.669798 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.670278 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.671362 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.673248 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.674171 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.674318 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.674462 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.675597 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.675834 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 11:11:44.677140 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.668889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:39453 (local address 127.8.181.3:36645)
0925 11:11:44.669293 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.669322 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.669476 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.669826 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.669830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.669856 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.670134 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.670146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.671513 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.671519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.673081 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.673094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.674661 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.674669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.674865 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.674912 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.676474 (+ 1562us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.676707 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"spinlock_wait_cycles":25344}
I20260925 11:11:44.679037 28552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "15f1b197211a453f863126cafbfee5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" is_pre_election: true
I20260925 11:11:44.679780 28552 raft_consensus.cc:2454] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15f1b197211a453f863126cafbfee5c3 in term 0.
I20260925 11:11:44.682930 28619 leader_election.cc:302] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15f1b197211a453f863126cafbfee5c3, fd42a5860eb443efbd7ceda935a4d6aa; no voters:
I20260925 11:11:44.683560 28757 raft_consensus.cc:2781] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:44.683730 28757 raft_consensus.cc:482] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:44.683840 28757 raft_consensus.cc:3037] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:44.688345 28757 raft_consensus.cc:504] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.688959 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.669113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55495 (local address 127.8.181.1:41751)
0925 11:11:44.669327 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.669333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.669843 (+ 510us) server_negotiation.cc:407] Connection header received
0925 11:11:44.670054 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.670062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.670140 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.670239 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.670840 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.671639 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.674061 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.675795 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.675963 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.676109 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.677328 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.677639 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:11:44.689051 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.674808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:39235 (local address 127.8.181.2:39453)
0925 11:11:44.677641 (+ 2833us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.677650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.678413 (+ 763us) server_negotiation.cc:407] Connection header received
0925 11:11:44.678769 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.678778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.678879 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.678956 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.679994 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.681570 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.685257 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.687508 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.687667 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.687824 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.688636 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.688780 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2685,"thread_start_us":13545,"threads_started":1}
I20260925 11:11:44.689051 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.674247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:39453 (local address 127.8.181.1:39235)
0925 11:11:44.678290 (+ 4043us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.678317 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.678569 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.679195 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.679203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.679237 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.679783 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.679799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.681826 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.681836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.684979 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.685003 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.688107 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.688114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.688368 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.688409 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.688515 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.688679 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":3694,"threads_started":1}
I20260925 11:11:44.689987 28757 leader_election.cc:288] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [CANDIDATE]: Term 1 election: Requested vote from peers 67965fc339d547ae9e8172b4249ff2c9 (127.8.181.1:41751), fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453)
I20260925 11:11:44.690454 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.668901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:41751 (local address 127.8.181.3:55495)
0925 11:11:44.669734 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.669751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.669901 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.670268 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.670272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.670296 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.670691 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.670703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.671800 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.671805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.673836 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.673851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.676778 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.676787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.677074 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.677133 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.689360 (+ 12227us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.689525 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":59,"thread_start_us":889,"threads_started":1}
I20260925 11:11:44.690447 28552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "67965fc339d547ae9e8172b4249ff2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" is_pre_election: true
I20260925 11:11:44.690851 28552 raft_consensus.cc:2454] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67965fc339d547ae9e8172b4249ff2c9 in term 0.
I20260925 11:11:44.690447 28551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "15f1b197211a453f863126cafbfee5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a5860eb443efbd7ceda935a4d6aa"
I20260925 11:11:44.691045 28551 raft_consensus.cc:3037] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:44.691676 28417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "15f1b197211a453f863126cafbfee5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67965fc339d547ae9e8172b4249ff2c9"
I20260925 11:11:44.691709 28418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "15f1b197211a453f863126cafbfee5c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67965fc339d547ae9e8172b4249ff2c9" is_pre_election: true
I20260925 11:11:44.692181 28418 raft_consensus.cc:2454] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15f1b197211a453f863126cafbfee5c3 in term 0.
I20260925 11:11:44.692813 28351 leader_election.cc:302] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67965fc339d547ae9e8172b4249ff2c9, fd42a5860eb443efbd7ceda935a4d6aa; no voters:
I20260925 11:11:44.693509 28758 raft_consensus.cc:2781] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:44.693653 28758 raft_consensus.cc:482] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:44.693763 28758 raft_consensus.cc:3037] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:44.694546 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.675255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:52939 (local address 127.8.181.3:37985)
0925 11:11:44.675504 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.675512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.686982 (+ 11470us) server_negotiation.cc:407] Connection header received
0925 11:11:44.687496 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.687502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.687577 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.687636 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:44.688541 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.689606 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.692250 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.693516 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.693613 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.693735 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.694141 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.694269 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:11:44.694646 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.674695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:37985 (local address 127.8.181.1:52939)
0925 11:11:44.686859 (+ 12164us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:44.686889 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:44.687322 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:44.687814 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:44.687819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:44.687844 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:44.688342 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.688355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.689783 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.689789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.692047 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:44.692063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.693845 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.693850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:44.693971 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.693998 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 11:11:44.694117 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 11:11:44.694260 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11979,"thread_start_us":12293,"threads_started":1}
I20260925 11:11:44.694643 28551 raft_consensus.cc:2454] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15f1b197211a453f863126cafbfee5c3 in term 1.
I20260925 11:11:44.695201 28619 leader_election.cc:302] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15f1b197211a453f863126cafbfee5c3, fd42a5860eb443efbd7ceda935a4d6aa; no voters: 67965fc339d547ae9e8172b4249ff2c9
I20260925 11:11:44.695605 28686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "67965fc339d547ae9e8172b4249ff2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15f1b197211a453f863126cafbfee5c3" is_pre_election: true
I20260925 11:11:44.695755 28757 raft_consensus.cc:2781] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:44.696306 28757 raft_consensus.cc:686] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 1 LEADER]: Becoming Leader. State: Replica: 15f1b197211a453f863126cafbfee5c3, State: Running, Role: LEADER
I20260925 11:11:44.696874 28757 consensus_queue.cc:237] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.697048 28758 raft_consensus.cc:504] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } }
I20260925 11:11:44.697805 28758 leader_election.cc:288] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [CANDIDATE]: Term 1 election: Requested vote from peers fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453), 15f1b197211a453f863126cafbfee5c3 (127.8.181.3:37985)
I20260925 11:11:44.698273 28551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "67965fc339d547ae9e8172b4249ff2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a5860eb443efbd7ceda935a4d6aa"
I20260925 11:11:44.698321 28685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "855f6b73e0a94a12ae8d271233f16031" candidate_uuid: "67965fc339d547ae9e8172b4249ff2c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15f1b197211a453f863126cafbfee5c3"
I20260925 11:11:44.698642 28551 raft_consensus.cc:2379] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 67965fc339d547ae9e8172b4249ff2c9 in current term 1: Already voted for candidate 15f1b197211a453f863126cafbfee5c3 in this term.
I20260925 11:11:44.699410 28351 leader_election.cc:302] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 67965fc339d547ae9e8172b4249ff2c9; no voters: 15f1b197211a453f863126cafbfee5c3, fd42a5860eb443efbd7ceda935a4d6aa
I20260925 11:11:44.699795 28758 raft_consensus.cc:2726] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 11:11:44.703176 28274 catalog_manager.cc:5495] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15f1b197211a453f863126cafbfee5c3 (127.8.181.3). New cstate: current_term: 1 leader_uuid: "15f1b197211a453f863126cafbfee5c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15f1b197211a453f863126cafbfee5c3" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 37985 } health_report { overall_health: HEALTHY } } }
I20260925 11:11:44.739989 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.734566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.8.181.3:37985)
0925 11:11:44.734730 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.734734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.734831 (+ 97us) server_negotiation.cc:407] Connection header received
0925 11:11:44.735002 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.735005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.735110 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.735245 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:44.735573 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.736301 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.737642 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.738132 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.738228 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.738809 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:44.739458 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:44.739575 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.739678 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.739746 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 11:11:44.748134 28417 raft_consensus.cc:1261] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 1 FOLLOWER]: Refusing update from remote peer 15f1b197211a453f863126cafbfee5c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:44.748134 28551 raft_consensus.cc:1261] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 1 FOLLOWER]: Refusing update from remote peer 15f1b197211a453f863126cafbfee5c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:44.749051 28763 consensus_queue.cc:1035] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd42a5860eb443efbd7ceda935a4d6aa" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:44.749931 28757 consensus_queue.cc:1035] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67965fc339d547ae9e8172b4249ff2c9" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 41751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:44.780968 28273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:44.783025 28273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:44.787868 28273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71f28e01bc8f4da7838e6e3c2e906946: Sending DeleteTablet for 3 replicas of tablet 855f6b73e0a94a12ae8d271233f16031
I20260925 11:11:44.789199 28398 tablet_service.cc:1514] Processing DeleteTablet for tablet 855f6b73e0a94a12ae8d271233f16031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:44 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:58434
I20260925 11:11:44.789388 28532 tablet_service.cc:1514] Processing DeleteTablet for tablet 855f6b73e0a94a12ae8d271233f16031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:44 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:47690
I20260925 11:11:44.789355 28666 tablet_service.cc:1514] Processing DeleteTablet for tablet 855f6b73e0a94a12ae8d271233f16031 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:44 UTC) from {username='kudu', principal='kudu/127.8.181.62@KRBTEST.COM'} at 127.0.0.1:58460
I20260925 11:11:44.794041 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28335
I20260925 11:11:44.802026 28778 tablet_replica.cc:330] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: stopping tablet replica
I20260925 11:11:44.802449 28776 tablet_replica.cc:330] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: stopping tablet replica
I20260925 11:11:44.802677 28778 raft_consensus.cc:2229] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:44.803102 28776 raft_consensus.cc:2229] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:44.803496 28778 raft_consensus.cc:2258] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:44.803805 28776 raft_consensus.cc:2258] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:44.806129 28776 ts_tablet_manager.cc:1905] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:44.809495 28778 ts_tablet_manager.cc:1905] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:44.812927 28777 tablet_replica.cc:330] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: stopping tablet replica
Sep 25 11:11:44 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.1@KRBTEST.COM
I20260925 11:11:44.813546 28777 raft_consensus.cc:2229] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:44.814108 28777 raft_consensus.cc:2258] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:44.815682 28776 ts_tablet_manager.cc:1918] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:44.815927 28776 log.cc:1196] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/855f6b73e0a94a12ae8d271233f16031
I20260925 11:11:44.816694 28776 ts_tablet_manager.cc:1939] T 855f6b73e0a94a12ae8d271233f16031 P 67965fc339d547ae9e8172b4249ff2c9: Deleting consensus metadata
I20260925 11:11:44.817364 28778 ts_tablet_manager.cc:1918] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:44.817549 28778 log.cc:1196] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/855f6b73e0a94a12ae8d271233f16031
I20260925 11:11:44.818131 28778 ts_tablet_manager.cc:1939] T 855f6b73e0a94a12ae8d271233f16031 P 15f1b197211a453f863126cafbfee5c3: Deleting consensus metadata
I20260925 11:11:44.819205 28259 catalog_manager.cc:4841] TS 67965fc339d547ae9e8172b4249ff2c9 (127.8.181.1:41751): tablet 855f6b73e0a94a12ae8d271233f16031 (table test-table [id=20878f6fde694f23b2a2f926fa56c52e]) successfully deleted
I20260925 11:11:44.820282 28260 catalog_manager.cc:4841] TS 15f1b197211a453f863126cafbfee5c3 (127.8.181.3:37985): tablet 855f6b73e0a94a12ae8d271233f16031 (table test-table [id=20878f6fde694f23b2a2f926fa56c52e]) successfully deleted
I20260925 11:11:44.821808 28777 ts_tablet_manager.cc:1905] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:44.824182 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.795325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.8.181.1:41751)
0925 11:11:44.795526 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.795533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.797857 (+ 2324us) server_negotiation.cc:407] Connection header received
0925 11:11:44.798084 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.798091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.798216 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.798323 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:44.803714 (+ 5391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.805000 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.807052 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.807730 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.807849 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.815888 (+ 8039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:44.815907 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:44.815934 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:44.815987 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:44.820494 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.821473 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.821482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.821487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.821674 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.822127 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.822133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.822137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.822681 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:44.822987 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.823335 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.823587 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 11:11:44.824926 28458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:44.829684 28777 ts_tablet_manager.cc:1918] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:44.829909 28777 log.cc:1196] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/855f6b73e0a94a12ae8d271233f16031
I20260925 11:11:44.830547 28777 ts_tablet_manager.cc:1939] T 855f6b73e0a94a12ae8d271233f16031 P fd42a5860eb443efbd7ceda935a4d6aa: Deleting consensus metadata
I20260925 11:11:44.832600 28258 catalog_manager.cc:4841] TS fd42a5860eb443efbd7ceda935a4d6aa (127.8.181.2:39453): tablet 855f6b73e0a94a12ae8d271233f16031 (table test-table [id=20878f6fde694f23b2a2f926fa56c52e]) successfully deleted
I20260925 11:11:44.957500 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28335
I20260925 11:11:44.976663 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28469
Sep 25 11:11:44 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.2@KRBTEST.COM
I20260925 11:11:44.995239 28748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:44.977662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.8.181.2:39453)
0925 11:11:44.977853 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:44.977856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:44.977874 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:11:44.978012 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:44.978016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:44.978079 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:44.978165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:44.979843 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.980606 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.982558 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:44.982965 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:44.983067 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:44.988626 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:44.988644 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:44.988660 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:44.988707 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:44.991766 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.992570 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.992578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.992580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.992712 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:44.993247 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:44.993253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:44.993256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:44.993735 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:44.994058 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:44.994462 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 11:11:44.994869 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:11:44.995913 28592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:45.088523 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28469
I20260925 11:11:45.112417 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28603
Sep 25 11:11:45 dist-test-slave-6042 krb5kdc[28229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.3@KRBTEST.COM
I20260925 11:11:45.132722 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:45.113274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.8.181.3:37985)
0925 11:11:45.113468 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:45.113472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:45.113563 (+ 91us) server_negotiation.cc:407] Connection header received
0925 11:11:45.113766 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:45.113770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:45.113862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:45.113942 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:45.115865 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:45.116630 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:45.118593 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:45.119088 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:45.119188 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:45.124848 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:45.124864 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:45.124885 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:45.124958 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:45.129455 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:45.130253 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:45.130268 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:45.130273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:45.130458 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:45.130870 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:45.130885 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:45.130889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:45.131458 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:45.131777 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:45.132075 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 11:11:45.132345 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:11:45.133510 28726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:45.260739 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28603
I20260925 11:11:45.281288 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28242
I20260925 11:11:45.282368 28304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:45.395467 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28242
2026-09-25T11:11:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T11:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:45Z Disabled control of system clock
I20260925 11:11:45.475654 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41217
--webserver_interface=127.8.181.62
--webserver_port=0
--builtin_ntp_servers=127.8.181.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:41217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:45.819763 28787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:45.820077 28787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:45.820236 28787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:45.827644 28787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:45.827765 28787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:45.827854 28787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:45.827958 28787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:45.828033 28787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:45.837447 28787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:41217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:45.838958 28787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:45.840794 28787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:45.853253 28792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:45.853314 28793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:45.853436 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:45.854663 28787 server_base.cc:1034] running on GCE node
I20260925 11:11:45.855613 28787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:45.857187 28787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:45.858503 28787 hybrid_clock.cc:648] HybridClock initialized: now 1790334705858406 us; error 52 us; skew 500 ppm
I20260925 11:11:45.859007 28787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:45.867697 28787 webserver.cc:460] Webserver started at http://127.8.181.62:41589/ using document root <none> and password file <none>
I20260925 11:11:45.868542 28787 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:45.868693 28787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:45.869042 28787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:45.872222 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "005f0fd0839f4776a60f2d3c6aee6bb1"
format_stamp: "Formatted at 2026-09-25 11:11:45 on dist-test-slave-6042"
server_key: "6dbee9c83edcaf4f12c5ca8b383ceab0"
server_key_iv: "0ec571f23daba51f6c2b13e5ddd61b49"
server_key_version: "encryptionkey@0"
I20260925 11:11:45.873063 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "005f0fd0839f4776a60f2d3c6aee6bb1"
format_stamp: "Formatted at 2026-09-25 11:11:45 on dist-test-slave-6042"
server_key: "6dbee9c83edcaf4f12c5ca8b383ceab0"
server_key_iv: "0ec571f23daba51f6c2b13e5ddd61b49"
server_key_version: "encryptionkey@0"
I20260925 11:11:45.878643 28787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 11:11:45.882300 28801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:45.883853 28787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:45.884066 28787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "005f0fd0839f4776a60f2d3c6aee6bb1"
format_stamp: "Formatted at 2026-09-25 11:11:45 on dist-test-slave-6042"
server_key: "6dbee9c83edcaf4f12c5ca8b383ceab0"
server_key_iv: "0ec571f23daba51f6c2b13e5ddd61b49"
server_key_version: "encryptionkey@0"
I20260925 11:11:45.884338 28787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:45.900556 28787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:45.901465 28787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:45.901774 28787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:45.921033 28787 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:41217
I20260925 11:11:45.921061 28852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:41217 every 8 connection(s)
I20260925 11:11:45.922516 28787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:45.926417 28853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:45.932837 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28787
I20260925 11:11:45.933033 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:45.933586 8916 external_mini_cluster.cc:1428] Setting key 4794c3e214f6856538efe0a11216c09a
I20260925 11:11:45.938098 28853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: Bootstrap starting.
I20260925 11:11:45.941354 28853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:45.942415 28853 log.cc:824] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:45.945555 28853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: No bootstrap required, opened a new log
I20260925 11:11:45.947811 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:45.935743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.8.181.62:41217)
0925 11:11:45.936745 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:45.936765 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:45.936805 (+ 40us) server_negotiation.cc:407] Connection header received
0925 11:11:45.938683 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:45.938707 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:45.939208 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:45.939633 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:45.940205 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:45.941495 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:45.943017 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:45.943771 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:45.943986 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:45.945231 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:45.945239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:45.945254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:45.945914 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:45.946038 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:45.946064 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:45.946206 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:45.946428 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 11:11:45.946824 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":338,"threads_started":1}
I20260925 11:11:45.953754 28853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } }
I20260925 11:11:45.954206 28853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:45.954315 28853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 005f0fd0839f4776a60f2d3c6aee6bb1, State: Initialized, Role: FOLLOWER
I20260925 11:11:45.955003 28853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } }
I20260925 11:11:45.955271 28853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:45.955404 28853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:45.955601 28853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:45.957886 28853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } }
I20260925 11:11:45.958436 28853 leader_election.cc:302] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 005f0fd0839f4776a60f2d3c6aee6bb1; no voters:
I20260925 11:11:45.959128 28853 leader_election.cc:288] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:45.959291 28858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:45.960014 28858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 005f0fd0839f4776a60f2d3c6aee6bb1, State: Running, Role: LEADER
I20260925 11:11:45.960690 28858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } }
I20260925 11:11:45.961340 28853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:45.963909 28860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 005f0fd0839f4776a60f2d3c6aee6bb1. Latest consensus state: current_term: 1 leader_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } } }
I20260925 11:11:45.963999 28859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "005f0fd0839f4776a60f2d3c6aee6bb1" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41217 } } }
I20260925 11:11:45.964576 28860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:45.964607 28859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1 [sys.catalog]: This master's current role is: LEADER
W20260925 11:11:45.972935 28874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:11:45.973070 28874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:11:45.973235 28867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:45.975487 28867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:45.984730 28867 catalog_manager.cc:1347] Generated new cluster ID: 56591c493c844129b2513eadc44176d1
I20260925 11:11:45.984866 28867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:46.007272 28867 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:46.009270 28867 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:46.025148 28867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: Generated new TSK 0
I20260925 11:11:46.026139 28867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:46.038056 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.1:0
--local_ip_for_outbound_sockets=127.8.181.1
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--builtin_ntp_servers=127.8.181.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 11:11:46.427372 28877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:46.427672 28877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:46.427758 28877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:46.435945 28877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:46.436054 28877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:46.436167 28877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.1
I20260925 11:11:46.445919 28877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.181.1
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.1
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:46.447343 28877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:46.449220 28877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:46.463377 28882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:46.463399 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:46.463496 28883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:46.464414 28877 server_base.cc:1034] running on GCE node
I20260925 11:11:46.465077 28877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:46.466297 28877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:46.467569 28877 hybrid_clock.cc:648] HybridClock initialized: now 1790334706467493 us; error 61 us; skew 500 ppm
I20260925 11:11:46.467952 28877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:46.470583 28877 webserver.cc:460] Webserver started at http://127.8.181.1:46643/ using document root <none> and password file <none>
I20260925 11:11:46.471364 28877 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:46.471508 28877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:46.471853 28877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:46.474949 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data/instance:
uuid: "8d73fbc3824b436bbf9f21d6501b9289"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "307dfa93020742c3620933bb59d64e2f"
server_key_iv: "61756e1438784635aaf9e556ed932ef1"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.475802 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance:
uuid: "8d73fbc3824b436bbf9f21d6501b9289"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "307dfa93020742c3620933bb59d64e2f"
server_key_iv: "61756e1438784635aaf9e556ed932ef1"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.481178 28877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:46.485170 28891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:46.486749 28877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:46.486944 28877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal
uuid: "8d73fbc3824b436bbf9f21d6501b9289"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "307dfa93020742c3620933bb59d64e2f"
server_key_iv: "61756e1438784635aaf9e556ed932ef1"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.487226 28877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:46.504945 28877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:46.505856 28877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:46.506170 28877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:46.507304 28877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:46.508878 28877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:46.509011 28877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:46.509138 28877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:46.509229 28877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:46.546825 28877 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.1:35947
I20260925 11:11:46.546890 29003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.1:35947 every 8 connection(s)
I20260925 11:11:46.548249 28877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/data/info.pb
I20260925 11:11:46.551211 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 28877
I20260925 11:11:46.551390 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal/instance
I20260925 11:11:46.551834 8916 external_mini_cluster.cc:1428] Setting key 1a57d0b9282d68e94823199173fc6405
I20260925 11:11:46.554530 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.2:0
--local_ip_for_outbound_sockets=127.8.181.2
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--builtin_ntp_servers=127.8.181.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:46.561355 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:46.550998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.1:53971 (local address 127.8.181.62:41217)
0925 11:11:46.551472 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:46.551479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:46.552841 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 11:11:46.554679 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:46.554685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:46.554776 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:46.554893 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:46.555975 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:46.556687 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:46.558299 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:46.558771 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:46.558916 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:46.560393 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:46.560398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:46.560401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:46.560422 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:46.560464 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:46.560476 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:46.560608 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:46.560853 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 11:11:46.560962 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260925 11:11:46.562633 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:46.551591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41217 (local address 127.8.181.1:53971)
0925 11:11:46.552678 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:46.552735 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:46.554354 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:46.555140 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:46.555149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:46.555237 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:46.555803 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:46.555819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:46.556861 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:46.556866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:46.558134 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:46.558146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:46.559014 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:46.559021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:46.559269 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:46.560022 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:46.560025 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:46.560066 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:46.560157 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:46.560618 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:46.560632 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:46.560642 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 11:11:46.560842 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 11:11:46.561289 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":356,"threads_started":1}
I20260925 11:11:46.564237 29004 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41217
I20260925 11:11:46.564603 29004 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:46.565653 29004 heartbeater.cc:511] Master 127.8.181.62:41217 requested a full tablet report, sending...
I20260925 11:11:46.568069 28818 ts_manager.cc:194] Registered new tserver with Master: 8d73fbc3824b436bbf9f21d6501b9289 (127.8.181.1:35947)
I20260925 11:11:46.569609 28818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.1:53971
W20260925 11:11:46.899348 29008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:46.899706 29008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:46.899876 29008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:46.907550 29008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:46.907718 29008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:46.907908 29008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.2
I20260925 11:11:46.917654 29008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.181.2
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.2
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:46.919220 29008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:46.921178 29008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:46.934547 29013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:46.934553 29014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:46.935273 29008 server_base.cc:1034] running on GCE node
W20260925 11:11:46.935750 29016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:46.936435 29008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:46.937579 29008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:46.938804 29008 hybrid_clock.cc:648] HybridClock initialized: now 1790334706938732 us; error 93 us; skew 500 ppm
I20260925 11:11:46.939146 29008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:46.941591 29008 webserver.cc:460] Webserver started at http://127.8.181.2:40863/ using document root <none> and password file <none>
I20260925 11:11:46.942329 29008 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:46.942512 29008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:46.942850 29008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:46.945803 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data/instance:
uuid: "3df10ce377864bcfb7a1f3f72c9d561d"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "fd72292f90f6ae15b5b4ee20442143bf"
server_key_iv: "89f05e2c8e2afa35445a27b30380ee36"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.946750 29008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance:
uuid: "3df10ce377864bcfb7a1f3f72c9d561d"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "fd72292f90f6ae15b5b4ee20442143bf"
server_key_iv: "89f05e2c8e2afa35445a27b30380ee36"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.952677 29008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:46.956525 29022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:46.958051 29008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:11:46.958348 29008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal
uuid: "3df10ce377864bcfb7a1f3f72c9d561d"
format_stamp: "Formatted at 2026-09-25 11:11:46 on dist-test-slave-6042"
server_key: "fd72292f90f6ae15b5b4ee20442143bf"
server_key_iv: "89f05e2c8e2afa35445a27b30380ee36"
server_key_version: "encryptionkey@0"
I20260925 11:11:46.958673 29008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:46.977948 29008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:46.978924 29008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:46.979218 29008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:46.980264 29008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:46.981822 29008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:46.981947 29008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:46.982084 29008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:46.982174 29008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:47.016949 29008 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.2:37237
I20260925 11:11:47.017009 29134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.2:37237 every 8 connection(s)
I20260925 11:11:47.018374 29008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/data/info.pb
I20260925 11:11:47.023187 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29008
I20260925 11:11:47.023366 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal/instance
I20260925 11:11:47.023823 8916 external_mini_cluster.cc:1428] Setting key d7580305badc843f9f9ec40a6e0b6995
I20260925 11:11:47.027292 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.181.3:0
--local_ip_for_outbound_sockets=127.8.181.3
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--builtin_ntp_servers=127.8.181.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 11:11:47.033615 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.021318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:48729 (local address 127.8.181.62:41217)
0925 11:11:47.021547 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.021552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.023254 (+ 1702us) server_negotiation.cc:407] Connection header received
0925 11:11:47.025543 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.025552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.025668 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.025782 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:47.027325 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.028558 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.030209 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.030991 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.031195 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.032625 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:47.032636 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:47.032644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:47.032673 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:47.032719 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:47.032738 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:47.032834 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.033106 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.033226 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 11:11:47.034614 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.021866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41217 (local address 127.8.181.2:48729)
0925 11:11:47.023064 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.023105 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.025196 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.026172 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.026187 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.026305 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:47.027043 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.027063 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.028744 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.028748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.030042 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.030056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.031263 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.031268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.031507 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.032258 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:47.032260 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:47.032303 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:47.032394 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:47.032864 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:47.032874 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:47.032881 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.033058 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.033378 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260925 11:11:47.036393 29135 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41217
I20260925 11:11:47.036753 29135 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:47.037721 29135 heartbeater.cc:511] Master 127.8.181.62:41217 requested a full tablet report, sending...
I20260925 11:11:47.039364 28818 ts_manager.cc:194] Registered new tserver with Master: 3df10ce377864bcfb7a1f3f72c9d561d (127.8.181.2:37237)
I20260925 11:11:47.040262 28818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.2:48729
W20260925 11:11:47.367014 29138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:47.367317 29138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:47.367458 29138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:47.375732 29138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:47.375847 29138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:47.376036 29138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.181.3
I20260925 11:11:47.385516 29138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.181.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.181.3
--webserver_port=0
--tserver_master_addrs=127.8.181.62:41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.181.3
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:47.386917 29138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:47.388813 29138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:47.402793 29143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:47.402783 29144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:47.403003 29146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:47.403891 29138 server_base.cc:1034] running on GCE node
I20260925 11:11:47.404518 29138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:47.405496 29138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:47.406711 29138 hybrid_clock.cc:648] HybridClock initialized: now 1790334707406661 us; error 37 us; skew 500 ppm
I20260925 11:11:47.407048 29138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 11:11:47.409497 29138 webserver.cc:460] Webserver started at http://127.8.181.3:45045/ using document root <none> and password file <none>
I20260925 11:11:47.410236 29138 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:47.410385 29138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:47.410763 29138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:47.413750 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data/instance:
uuid: "ebf942c918e04834891865ccf3ceb8e9"
format_stamp: "Formatted at 2026-09-25 11:11:47 on dist-test-slave-6042"
server_key: "99defc700c21333b0ae568ed49d4d388"
server_key_iv: "7de99f2efa5dedbc931c1b39548281e6"
server_key_version: "encryptionkey@0"
I20260925 11:11:47.414667 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance:
uuid: "ebf942c918e04834891865ccf3ceb8e9"
format_stamp: "Formatted at 2026-09-25 11:11:47 on dist-test-slave-6042"
server_key: "99defc700c21333b0ae568ed49d4d388"
server_key_iv: "7de99f2efa5dedbc931c1b39548281e6"
server_key_version: "encryptionkey@0"
I20260925 11:11:47.420017 29138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 11:11:47.423679 29152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:47.425117 29138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 11:11:47.425311 29138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal
uuid: "ebf942c918e04834891865ccf3ceb8e9"
format_stamp: "Formatted at 2026-09-25 11:11:47 on dist-test-slave-6042"
server_key: "99defc700c21333b0ae568ed49d4d388"
server_key_iv: "7de99f2efa5dedbc931c1b39548281e6"
server_key_version: "encryptionkey@0"
I20260925 11:11:47.425547 29138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:47.437963 29138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:47.438962 29138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:47.439248 29138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:47.440220 29138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 11:11:47.441681 29138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 11:11:47.441807 29138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:47.441932 29138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 11:11:47.442019 29138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:47.478245 29138 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.3:38355
I20260925 11:11:47.478302 29264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.3:38355 every 8 connection(s)
I20260925 11:11:47.479704 29138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/data/info.pb
I20260925 11:11:47.487265 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29138
I20260925 11:11:47.487471 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal/instance
I20260925 11:11:47.488018 8916 external_mini_cluster.cc:1428] Setting key b3f4d65a260b191120cf42c763fef9a2
I20260925 11:11:47.494001 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.482522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.3:55699 (local address 127.8.181.62:41217)
0925 11:11:47.482726 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.482730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.484189 (+ 1459us) server_negotiation.cc:407] Connection header received
0925 11:11:47.486072 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.486078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.486178 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.486294 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:47.487787 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.488836 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.491090 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.491577 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.491759 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.493285 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:47.493290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:47.493293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:47.493315 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:47.493348 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:47.493360 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:47.493429 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.493661 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.493741 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 11:11:47.495117 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.483025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:41217 (local address 127.8.181.3:55699)
0925 11:11:47.484012 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.484051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.485769 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.486597 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.486605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.486706 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:11:47.487567 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.487594 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.489059 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.489064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.490934 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.490949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.491812 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.491817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.492081 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.492920 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 11:11:47.492922 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:11:47.493001 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 11:11:47.493087 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:11:47.493446 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:11:47.493454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:11:47.493461 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.493611 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.493914 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":298,"threads_started":1}
I20260925 11:11:47.496663 29265 heartbeater.cc:347] Connected to a master server at 127.8.181.62:41217
I20260925 11:11:47.496941 29265 heartbeater.cc:464] Registering TS with master...
I20260925 11:11:47.497767 29265 heartbeater.cc:511] Master 127.8.181.62:41217 requested a full tablet report, sending...
I20260925 11:11:47.499351 28818 ts_manager.cc:194] Registered new tserver with Master: ebf942c918e04834891865ccf3ceb8e9 (127.8.181.3:38355)
I20260925 11:11:47.500165 28818 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.181.3:55699
I20260925 11:11:47.503397 8916 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 11:11:47.513561 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.507716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.8.181.62:41217)
0925 11:11:47.507987 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.507992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.508406 (+ 414us) server_negotiation.cc:407] Connection header received
0925 11:11:47.508579 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.508584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.508679 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.508774 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:47.509250 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.509981 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.511392 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.511792 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.511891 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.512971 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:47.512976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:47.512979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:47.512998 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:47.513032 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:47.513043 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:47.513117 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.513236 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.513319 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 11:11:47.517678 28818 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 11:11:47.524763 28818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 11:11:47.561939 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.537861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:38355 (local address 127.0.0.1:54204)
0925 11:11:47.550262 (+ 12401us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.550316 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.551412 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.553369 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.553377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.553411 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:47.553814 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.553826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.555833 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.555840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.558427 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.558445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.560795 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.560800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.561057 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.561123 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.561322 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.561593 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12110,"thread_start_us":18114,"threads_started":1}
I20260925 11:11:47.563261 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.539012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.8.181.3:38355)
0925 11:11:47.546018 (+ 7006us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.546028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.550459 (+ 4431us) server_negotiation.cc:407] Connection header received
0925 11:11:47.552526 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.552536 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.552831 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.553011 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:47.554023 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.555641 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.558723 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.560197 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.560308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.560615 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.561393 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.561570 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":854,"threads_started":1}
I20260925 11:11:47.563812 29277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.539752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:35947 (local address 127.0.0.1:38912)
0925 11:11:47.556325 (+ 16573us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.556359 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.556501 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.557075 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.557079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.557112 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:47.557486 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.557494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.559211 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.559215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.560705 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.560721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.562934 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.562940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.563157 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.563190 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.563310 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.563442 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16414,"mutex_wait_us":3,"thread_start_us":14735,"threads_started":1}
I20260925 11:11:47.563980 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.539996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.8.181.1:35947)
0925 11:11:47.552195 (+ 12199us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.552204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.556489 (+ 4285us) server_negotiation.cc:407] Connection header received
0925 11:11:47.556652 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.556660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.556892 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.557089 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:47.557640 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.559035 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.560934 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.562518 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.562663 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.562992 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.563364 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.563519 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12014,"thread_start_us":12188,"threads_started":1}
I20260925 11:11:47.563987 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.538546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.8.181.2:37237)
0925 11:11:47.539108 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.539117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.553546 (+ 14429us) server_negotiation.cc:407] Connection header received
0925 11:11:47.554776 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.554788 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.555079 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.555286 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:47.556801 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.558361 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.561073 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.562515 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.562653 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.562909 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.563474 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.563661 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":749,"threads_started":1}
I20260925 11:11:47.564615 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.538268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.2:37237 (local address 127.0.0.1:42882)
0925 11:11:47.553352 (+ 15084us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.553382 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.554506 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.556037 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.556043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.556078 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:47.556581 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.556594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.558562 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.558569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.560886 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.560897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.563050 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.563057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.563259 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.563301 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.564181 (+ 880us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.564319 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14866,"mutex_wait_us":73,"thread_start_us":15328,"threads_started":1}
I20260925 11:11:47.568056 28939 tablet_service.cc:1467] Processing CreateTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 (DEFAULT_TABLE table=test-table [id=02f4d15afafa43f8b9180b69696651e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:47.567795 29200 tablet_service.cc:1467] Processing CreateTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 (DEFAULT_TABLE table=test-table [id=02f4d15afafa43f8b9180b69696651e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:47.568174 29070 tablet_service.cc:1467] Processing CreateTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 (DEFAULT_TABLE table=test-table [id=02f4d15afafa43f8b9180b69696651e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 11:11:47.569264 28939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e8e41d92e5475fac0f21c30acf6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:47.569348 29070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e8e41d92e5475fac0f21c30acf6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:47.569346 29200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59e8e41d92e5475fac0f21c30acf6901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:47.572924 29004 heartbeater.cc:503] Master 127.8.181.62:41217 was elected leader, sending a full tablet report...
I20260925 11:11:47.581460 29283 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Bootstrap starting.
I20260925 11:11:47.583693 29285 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Bootstrap starting.
I20260925 11:11:47.584530 29284 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Bootstrap starting.
I20260925 11:11:47.586826 29283 tablet_bootstrap.cc:654] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:47.587440 29285 tablet_bootstrap.cc:654] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:47.587925 29284 tablet_bootstrap.cc:654] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:47.588218 29283 log.cc:824] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:47.588583 29285 log.cc:824] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:47.589144 29284 log.cc:824] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:47.591764 29283 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: No bootstrap required, opened a new log
I20260925 11:11:47.591769 29285 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: No bootstrap required, opened a new log
I20260925 11:11:47.592056 29285 ts_tablet_manager.cc:1397] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 11:11:47.592056 29283 ts_tablet_manager.cc:1397] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260925 11:11:47.592424 29284 tablet_bootstrap.cc:492] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: No bootstrap required, opened a new log
I20260925 11:11:47.592702 29284 ts_tablet_manager.cc:1397] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 11:11:47.600360 29284 raft_consensus.cc:348] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.600768 29284 raft_consensus.cc:374] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:47.600906 29284 raft_consensus.cc:729] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf942c918e04834891865ccf3ceb8e9, State: Initialized, Role: FOLLOWER
I20260925 11:11:47.601557 29284 consensus_queue.cc:260] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.602705 29265 heartbeater.cc:503] Master 127.8.181.62:41217 was elected leader, sending a full tablet report...
I20260925 11:11:47.603035 29283 raft_consensus.cc:348] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.603440 29284 ts_tablet_manager.cc:1428] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260925 11:11:47.603479 29283 raft_consensus.cc:374] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:47.603571 29283 raft_consensus.cc:729] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d73fbc3824b436bbf9f21d6501b9289, State: Initialized, Role: FOLLOWER
I20260925 11:11:47.603811 29285 raft_consensus.cc:348] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.604279 29285 raft_consensus.cc:374] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:47.604208 29283 consensus_queue.cc:260] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.604416 29285 raft_consensus.cc:729] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df10ce377864bcfb7a1f3f72c9d561d, State: Initialized, Role: FOLLOWER
I20260925 11:11:47.605207 29285 consensus_queue.cc:260] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.607568 29283 ts_tablet_manager.cc:1428] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260925 11:11:47.608687 29135 heartbeater.cc:503] Master 127.8.181.62:41217 was elected leader, sending a full tablet report...
I20260925 11:11:47.609297 29285 ts_tablet_manager.cc:1428] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
W20260925 11:11:47.731439 29266 tablet.cc:2369] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:47.749850 29291 raft_consensus.cc:482] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 11:11:47.750313 29291 raft_consensus.cc:504] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.752147 29291 leader_election.cc:288] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebf942c918e04834891865ccf3ceb8e9 (127.8.181.3:38355), 8d73fbc3824b436bbf9f21d6501b9289 (127.8.181.1:35947)
I20260925 11:11:47.758937 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.752562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:46623 (local address 127.8.181.3:38355)
0925 11:11:47.752808 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.752818 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.753513 (+ 695us) server_negotiation.cc:407] Connection header received
0925 11:11:47.753755 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.753759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.753818 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.753918 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:47.754508 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.755247 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.757201 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.757977 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.758126 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.758234 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.758549 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.758680 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:11:47.758981 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.752795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.181.2:52377 (local address 127.8.181.1:35947)
0925 11:11:47.753060 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.753069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.753716 (+ 647us) server_negotiation.cc:407] Connection header received
0925 11:11:47.753921 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.753924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.753997 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.754102 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 11:11:47.754744 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.755432 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.757272 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.758037 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.758182 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.758282 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.758613 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.758739 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 11:11:47.759060 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.752401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.3:38355 (local address 127.8.181.2:46623)
0925 11:11:47.753381 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.753397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.753591 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.753935 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.753939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.753962 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:47.754326 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.754340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.755382 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.755389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.756999 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.757015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.758186 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.758193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.758366 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.758389 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.758515 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.758643 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":83,"thread_start_us":609,"threads_started":1}
I20260925 11:11:47.759106 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.752556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.1:35947 (local address 127.8.181.2:52377)
0925 11:11:47.753595 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0925 11:11:47.753610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 11:11:47.753763 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:11:47.754118 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:11:47.754122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:11:47.754148 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 11:11:47.754593 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.754603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.755573 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.755579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.757069 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:11:47.757081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.758227 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.758233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:11:47.758406 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.758428 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 11:11:47.758572 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 11:11:47.758693 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":42,"thread_start_us":721,"threads_started":1}
I20260925 11:11:47.760097 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e8e41d92e5475fac0f21c30acf6901" candidate_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf942c918e04834891865ccf3ceb8e9" is_pre_election: true
I20260925 11:11:47.760092 28959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e8e41d92e5475fac0f21c30acf6901" candidate_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d73fbc3824b436bbf9f21d6501b9289" is_pre_election: true
I20260925 11:11:47.760605 28959 raft_consensus.cc:2454] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df10ce377864bcfb7a1f3f72c9d561d in term 0.
I20260925 11:11:47.760612 29220 raft_consensus.cc:2454] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df10ce377864bcfb7a1f3f72c9d561d in term 0.
I20260925 11:11:47.761439 29024 leader_election.cc:302] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df10ce377864bcfb7a1f3f72c9d561d, ebf942c918e04834891865ccf3ceb8e9; no voters:
I20260925 11:11:47.761904 29291 raft_consensus.cc:2781] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 11:11:47.762071 29291 raft_consensus.cc:482] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 11:11:47.762195 29291 raft_consensus.cc:3037] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:47.765265 29291 raft_consensus.cc:504] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.765946 29291 leader_election.cc:288] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [CANDIDATE]: Term 1 election: Requested vote from peers ebf942c918e04834891865ccf3ceb8e9 (127.8.181.3:38355), 8d73fbc3824b436bbf9f21d6501b9289 (127.8.181.1:35947)
I20260925 11:11:47.766571 29220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e8e41d92e5475fac0f21c30acf6901" candidate_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebf942c918e04834891865ccf3ceb8e9"
I20260925 11:11:47.766654 28959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59e8e41d92e5475fac0f21c30acf6901" candidate_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d73fbc3824b436bbf9f21d6501b9289"
I20260925 11:11:47.766801 29220 raft_consensus.cc:3037] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:47.766906 28959 raft_consensus.cc:3037] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:47.769088 29220 raft_consensus.cc:2454] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df10ce377864bcfb7a1f3f72c9d561d in term 1.
I20260925 11:11:47.769098 28959 raft_consensus.cc:2454] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df10ce377864bcfb7a1f3f72c9d561d in term 1.
I20260925 11:11:47.769517 29024 leader_election.cc:302] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df10ce377864bcfb7a1f3f72c9d561d, ebf942c918e04834891865ccf3ceb8e9; no voters:
I20260925 11:11:47.769837 29291 raft_consensus.cc:2781] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:47.770354 29291 raft_consensus.cc:686] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 1 LEADER]: Becoming Leader. State: Replica: 3df10ce377864bcfb7a1f3f72c9d561d, State: Running, Role: LEADER
W20260925 11:11:47.770633 29136 tablet.cc:2369] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:47.770956 29291 consensus_queue.cc:237] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } }
I20260925 11:11:47.775494 28817 catalog_manager.cc:5495] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3df10ce377864bcfb7a1f3f72c9d561d (127.8.181.2). New cstate: current_term: 1 leader_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3df10ce377864bcfb7a1f3f72c9d561d" member_type: VOTER last_known_addr { host: "127.8.181.2" port: 37237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 } health_report { overall_health: UNKNOWN } } }
W20260925 11:11:47.800899 29005 tablet.cc:2369] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:11:47.805289 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.799770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.8.181.2:37237)
0925 11:11:47.800026 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.800031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.800050 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:11:47.800190 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.800194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.800291 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.800441 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:11:47.800853 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.801631 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.803045 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.803464 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.803561 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.804062 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:11:47.804624 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:11:47.804768 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.804849 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.804920 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 11:11:47.813863 28959 raft_consensus.cc:1261] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 1 FOLLOWER]: Refusing update from remote peer 3df10ce377864bcfb7a1f3f72c9d561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:47.813864 29220 raft_consensus.cc:1261] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 1 FOLLOWER]: Refusing update from remote peer 3df10ce377864bcfb7a1f3f72c9d561d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:11:47.814970 29294 consensus_queue.cc:1035] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d73fbc3824b436bbf9f21d6501b9289" member_type: VOTER last_known_addr { host: "127.8.181.1" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:47.815308 29291 consensus_queue.cc:1035] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebf942c918e04834891865ccf3ceb8e9" member_type: VOTER last_known_addr { host: "127.8.181.3" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:11:47.831710 29298 mvcc.cc:205] Tried to move back new op lower bound from 7333210963194744832 to 7333210963033968640. Current Snapshot: MvccSnapshot[applied={T|T < 7333210963194744832}]
I20260925 11:11:47.840476 28818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:47.840760 28818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 11:11:47.844591 28818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 005f0fd0839f4776a60f2d3c6aee6bb1: Sending DeleteTablet for 3 replicas of tablet 59e8e41d92e5475fac0f21c30acf6901
I20260925 11:11:47.845896 29200 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:47 UTC) from {username='slave'} at 127.0.0.1:54204
I20260925 11:11:47.845902 29070 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:47 UTC) from {username='slave'} at 127.0.0.1:42882
I20260925 11:11:47.846257 28939 tablet_service.cc:1514] Processing DeleteTablet for tablet 59e8e41d92e5475fac0f21c30acf6901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 11:11:47 UTC) from {username='slave'} at 127.0.0.1:38912
I20260925 11:11:47.847611 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28877
I20260925 11:11:47.861723 29308 tablet_replica.cc:330] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: stopping tablet replica
I20260925 11:11:47.862910 29308 raft_consensus.cc:2229] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:47.863687 29308 raft_consensus.cc:2258] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:47.866106 29308 ts_tablet_manager.cc:1905] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:47.873633 29308 ts_tablet_manager.cc:1918] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:47.873920 29308 log.cc:1196] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-0/wal/wals/59e8e41d92e5475fac0f21c30acf6901
I20260925 11:11:47.874639 29308 ts_tablet_manager.cc:1939] T 59e8e41d92e5475fac0f21c30acf6901 P 8d73fbc3824b436bbf9f21d6501b9289: Deleting consensus metadata
I20260925 11:11:47.876919 28804 catalog_manager.cc:4841] TS 8d73fbc3824b436bbf9f21d6501b9289 (127.8.181.1:35947): tablet 59e8e41d92e5475fac0f21c30acf6901 (table test-table [id=02f4d15afafa43f8b9180b69696651e5]) successfully deleted
I20260925 11:11:47.877827 29306 tablet_replica.cc:330] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: stopping tablet replica
I20260925 11:11:47.878100 29307 tablet_replica.cc:330] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: stopping tablet replica
I20260925 11:11:47.878382 29306 raft_consensus.cc:2229] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 11:11:47.878682 29307 raft_consensus.cc:2229] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 1 LEADER]: Raft consensus shutting down.
I20260925 11:11:47.879067 29306 raft_consensus.cc:2258] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:47.879434 29307 raft_consensus.cc:2258] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 11:11:47.881459 29307 ts_tablet_manager.cc:1905] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:47.881529 29306 ts_tablet_manager.cc:1905] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 11:11:47.883260 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:47.849210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.8.181.1:35947)
0925 11:11:47.849436 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:47.849446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:47.873621 (+ 24175us) server_negotiation.cc:407] Connection header received
0925 11:11:47.874366 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:47.874375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:47.874568 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:47.874670 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:47.875565 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.877079 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.879288 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:47.879951 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:47.880095 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:47.881551 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:47.881565 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:47.881581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:47.882331 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:47.882388 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:47.882412 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:47.882526 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:47.882766 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 11:11:47.882878 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:11:47.884572 28999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:47.888816 29306 ts_tablet_manager.cc:1918] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:47.889060 29306 log.cc:1196] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-2/wal/wals/59e8e41d92e5475fac0f21c30acf6901
I20260925 11:11:47.889966 29306 ts_tablet_manager.cc:1939] T 59e8e41d92e5475fac0f21c30acf6901 P ebf942c918e04834891865ccf3ceb8e9: Deleting consensus metadata
I20260925 11:11:47.890508 29307 ts_tablet_manager.cc:1918] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 11:11:47.890702 29307 log.cc:1196] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Deleting WAL directory at /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790334376327139-8916-0/minicluster-data/ts-1/wal/wals/59e8e41d92e5475fac0f21c30acf6901
I20260925 11:11:47.891306 29307 ts_tablet_manager.cc:1939] T 59e8e41d92e5475fac0f21c30acf6901 P 3df10ce377864bcfb7a1f3f72c9d561d: Deleting consensus metadata
I20260925 11:11:47.892803 28803 catalog_manager.cc:4841] TS ebf942c918e04834891865ccf3ceb8e9 (127.8.181.3:38355): tablet 59e8e41d92e5475fac0f21c30acf6901 (table test-table [id=02f4d15afafa43f8b9180b69696651e5]) successfully deleted
I20260925 11:11:47.893137 28802 catalog_manager.cc:4841] TS 3df10ce377864bcfb7a1f3f72c9d561d (127.8.181.2:37237): tablet 59e8e41d92e5475fac0f21c30acf6901 (table test-table [id=02f4d15afafa43f8b9180b69696651e5]) successfully deleted
I20260925 11:11:48.021222 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28877
I20260925 11:11:48.046061 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29008
I20260925 11:11:48.053967 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:48.046977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42892 (local address 127.8.181.2:37237)
0925 11:11:48.047212 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:48.047218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:48.047240 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:11:48.047310 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:48.047318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:48.047457 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:48.047581 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:48.048002 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:48.049212 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:48.050650 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:48.051133 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:48.051286 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:48.052433 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:48.052446 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:48.052461 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:48.053131 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:48.053184 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:48.053204 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:48.053329 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:48.053480 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 11:11:48.053584 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 11:11:48.054723 29130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:48.171113 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29008
I20260925 11:11:48.190143 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29138
I20260925 11:11:48.197336 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:48.190957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.8.181.3:38355)
0925 11:11:48.191185 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:48.191192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:48.191304 (+ 112us) server_negotiation.cc:407] Connection header received
0925 11:11:48.191523 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:48.191527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:48.191606 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:48.191716 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:48.192122 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:48.192837 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:48.194252 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:48.194705 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:48.194836 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:48.195977 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:48.195989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:48.196009 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:48.196582 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:48.196629 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:48.196648 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:48.196765 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:48.196906 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 11:11:48.197016 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 11:11:48.198117 29260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:48.288084 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29138
I20260925 11:11:48.306645 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 28787
I20260925 11:11:48.307765 28848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:48.420068 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 28787
2026-09-25T11:11:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3024 ms)
[----------] 6 tests from EncryptionPolicyTest (22661 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:48 dist-test-slave-6042 krb5kdc[29316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:48 dist-test-slave-6042 krb5kdc[29316](info): set up 2 sockets
Sep 25 11:11:48 dist-test-slave-6042 krb5kdc[29316](info): commencing operation
krb5kdc: starting...
W20260925 11:11:49.659317 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:49 dist-test-slave-6042 krb5kdc[29316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:11:49.939666 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:50.280364 29329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:50.280671 29329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:50.280831 29329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:50.288295 29329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:50.288427 29329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:50.288532 29329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:50.288643 29329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:50.288703 29329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:50.298951 29329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:50.300428 29329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:50.301812 29329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:50.313939 29337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:50.313963 29334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:50.314024 29335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:50.314913 29329 server_base.cc:1034] running on GCE node
I20260925 11:11:50.315722 29329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:50.317243 29329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:50.318550 29329 hybrid_clock.cc:648] HybridClock initialized: now 1790334710318447 us; error 55 us; skew 500 ppm
I20260925 11:11:50.318913 29329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:50 dist-test-slave-6042 krb5kdc[29316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:50.322172 29329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:11:50.323820 29329 webserver.cc:460] Webserver started at http://127.0.0.1:36327/ using document root <none> and password file <none>
I20260925 11:11:50.324559 29329 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:50.324704 29329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:50.325080 29329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:50.328182 29329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "c4d35d5f885e43e69d41b5de15736032"
format_stamp: "Formatted at 2026-09-25 11:11:50 on dist-test-slave-6042"
server_key: "8c434900a2c93d91857f7f71310005be"
server_key_iv: "e941b447e8dee89f2b87321d187fc5a2"
server_key_version: "encryptionkey@0"
I20260925 11:11:50.328949 29329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "c4d35d5f885e43e69d41b5de15736032"
format_stamp: "Formatted at 2026-09-25 11:11:50 on dist-test-slave-6042"
server_key: "8c434900a2c93d91857f7f71310005be"
server_key_iv: "e941b447e8dee89f2b87321d187fc5a2"
server_key_version: "encryptionkey@0"
I20260925 11:11:50.334268 29329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 11:11:50.337815 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:50.339234 29329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:50.339455 29329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "c4d35d5f885e43e69d41b5de15736032"
format_stamp: "Formatted at 2026-09-25 11:11:50 on dist-test-slave-6042"
server_key: "8c434900a2c93d91857f7f71310005be"
server_key_iv: "e941b447e8dee89f2b87321d187fc5a2"
server_key_version: "encryptionkey@0"
I20260925 11:11:50.339669 29329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:50.360821 29329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:50.365665 29329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:50.365993 29329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:50.387202 29329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46183
I20260925 11:11:50.387233 29395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46183 every 8 connection(s)
I20260925 11:11:50.388761 29329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:50.392578 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:50.397071 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29329
I20260925 11:11:50.397361 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:50.398070 8916 external_mini_cluster.cc:1428] Setting key a669632a88e317bbaf55555b1b2a2f94
I20260925 11:11:50.405500 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032: Bootstrap starting.
I20260925 11:11:50.409441 29396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:50.410781 29396 log.cc:824] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:50.414290 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032: No bootstrap required, opened a new log
Sep 25 11:11:50 dist-test-slave-6042 krb5kdc[29316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 11:11:50.427177 29396 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260925 11:11:50.427731 29396 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:50.427858 29396 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d35d5f885e43e69d41b5de15736032, State: Initialized, Role: FOLLOWER
I20260925 11:11:50.428241 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:50.400288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56457 (local address 127.0.0.1:46183)
0925 11:11:50.401182 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:50.401197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:50.401248 (+ 51us) server_negotiation.cc:407] Connection header received
0925 11:11:50.403828 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:50.403865 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:50.404396 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:50.404874 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:50.406437 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:50.408552 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:50.410045 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:50.411176 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:50.411402 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:50.416872 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:50.416885 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:50.416925 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:50.416991 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:50.422032 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:50.423306 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:50.423323 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:50.423334 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:50.423626 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:50.424299 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:50.424305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:50.424309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:50.425238 (+ 929us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:50.425611 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:50.426135 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 11:11:50.426585 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":388,"threads_started":1}
I20260925 11:11:50.428719 29396 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260925 11:11:50.429019 29396 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:50.429184 29396 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:50.429418 29396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:50.432533 29396 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260925 11:11:50.433094 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:11:50.433220 29396 leader_election.cc:302] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4d35d5f885e43e69d41b5de15736032; no voters:
I20260925 11:11:50.433947 29396 leader_election.cc:288] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:50.434055 29401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:50.436429 29396 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:50.439136 29401 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [term 1 LEADER]: Becoming Leader. State: Replica: c4d35d5f885e43e69d41b5de15736032, State: Running, Role: LEADER
I20260925 11:11:50.439867 29401 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } }
I20260925 11:11:50.442953 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4d35d5f885e43e69d41b5de15736032. Latest consensus state: current_term: 1 leader_uuid: "c4d35d5f885e43e69d41b5de15736032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } } }
I20260925 11:11:50.443383 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:50.443949 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4d35d5f885e43e69d41b5de15736032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4d35d5f885e43e69d41b5de15736032" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46183 } } }
I20260925 11:11:50.444216 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:50.455901 29412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:50.463119 29412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:50.468883 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:50.440270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57931 (local address 127.0.0.1:46183)
0925 11:11:50.441332 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:50.441342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:50.441372 (+ 30us) server_negotiation.cc:407] Connection header received
0925 11:11:50.441456 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:50.441465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:50.441575 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:50.444560 (+ 2985us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:50.444799 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:50.446649 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:50.449709 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:50.451546 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:50.452971 (+ 1425us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:50.456697 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:50.456704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:50.456709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:50.456754 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:50.462850 (+ 6096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:50.464480 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:50.464489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:50.464495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:50.464685 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:50.465450 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:50.465458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:50.465496 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:50.466088 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:50.467814 (+ 1726us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:50.468072 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 11:11:50.468326 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871}
I20260925 11:11:50.477214 29412 catalog_manager.cc:1347] Generated new cluster ID: 56d297ed9bd2485a8f33fe3072a959ca
I20260925 11:11:50.477370 29412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:50.493307 29412 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:50.495283 29412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:50.512567 29412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4d35d5f885e43e69d41b5de15736032: Generated new TSK 0
I20260925 11:11:50.513685 29412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:50.532442 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29329
I20260925 11:11:50.534565 29391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:50.640596 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29329
W20260925 11:11:50.658689 29314 connection.cc:535] client connection to 127.0.0.1:46183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46183): Connection reset by peer (error 104)
2026-09-25T11:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:50 dist-test-slave-6042 krb5kdc[29429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:50 dist-test-slave-6042 krb5kdc[29429](info): set up 2 sockets
Sep 25 11:11:50 dist-test-slave-6042 krb5kdc[29429](info): commencing operation
krb5kdc: starting...
W20260925 11:11:51.763550 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:51 dist-test-slave-6042 krb5kdc[29429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:11:52.039348 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:52.385231 29442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:52.385632 29442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:52.385804 29442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:52.393307 29442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:52.393500 29442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:52.393612 29442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:52.393728 29442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:52.393813 29442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:52.404086 29442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42869
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:52.405696 29442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:52.407097 29442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:52.419709 29447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:52.419766 29448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:52.419790 29450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:52.420729 29442 server_base.cc:1034] running on GCE node
I20260925 11:11:52.421669 29442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:52.423338 29442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:52.424603 29442 hybrid_clock.cc:648] HybridClock initialized: now 1790334712424562 us; error 49 us; skew 500 ppm
I20260925 11:11:52.425050 29442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:52 dist-test-slave-6042 krb5kdc[29429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:52.436295 29442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:11:52.438061 29442 webserver.cc:460] Webserver started at http://127.0.0.1:33693/ using document root <none> and password file <none>
I20260925 11:11:52.438934 29442 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:52.439102 29442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:52.439428 29442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:52.442648 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "0919132e6b5c4eea934c0facf88a2d06"
format_stamp: "Formatted at 2026-09-25 11:11:52 on dist-test-slave-6042"
server_key: "28cad5e5c2036f81839a791819480b1c"
server_key_iv: "1dc1c544af79208df37bf14e1c3ca8d1"
server_key_version: "encryptionkey@0"
I20260925 11:11:52.443473 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "0919132e6b5c4eea934c0facf88a2d06"
format_stamp: "Formatted at 2026-09-25 11:11:52 on dist-test-slave-6042"
server_key: "28cad5e5c2036f81839a791819480b1c"
server_key_iv: "1dc1c544af79208df37bf14e1c3ca8d1"
server_key_version: "encryptionkey@0"
I20260925 11:11:52.449030 29442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:11:52.453029 29457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:52.454672 29442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:52.454887 29442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "0919132e6b5c4eea934c0facf88a2d06"
format_stamp: "Formatted at 2026-09-25 11:11:52 on dist-test-slave-6042"
server_key: "28cad5e5c2036f81839a791819480b1c"
server_key_iv: "1dc1c544af79208df37bf14e1c3ca8d1"
server_key_version: "encryptionkey@0"
I20260925 11:11:52.455112 29442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:52.468467 29442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:52.472924 29442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:52.473246 29442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:52.493569 29442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42869
I20260925 11:11:52.493621 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42869 every 8 connection(s)
I20260925 11:11:52.495173 29442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:52.496872 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29442
I20260925 11:11:52.497061 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:52.497656 8916 external_mini_cluster.cc:1428] Setting key 02e0ffcfe82945aba9b0533233622136
I20260925 11:11:52.500211 29509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:52.512013 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:52.499733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51083 (local address 127.0.0.1:42869)
0925 11:11:52.500671 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:52.500683 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:52.500727 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:11:52.502439 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:52.502505 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:52.502942 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:52.503396 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:52.503988 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:52.505621 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:52.507071 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:52.507801 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:52.508021 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:52.509466 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:52.509476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:52.509493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:52.510112 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:52.510207 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:52.510235 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:52.510424 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:52.510742 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 11:11:52.510943 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":436,"threads_started":1}
I20260925 11:11:52.512495 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06: Bootstrap starting.
I20260925 11:11:52.516141 29509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:52.516368 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:11:52.517342 29509 log.cc:824] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:52.520566 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06: No bootstrap required, opened a new log
I20260925 11:11:52.531941 29512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:52.521237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51857 (local address 127.0.0.1:42869)
0925 11:11:52.521468 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:52.521478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:52.521952 (+ 474us) server_negotiation.cc:407] Connection header received
0925 11:11:52.522183 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:52.522186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:52.522250 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:52.522334 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:52.523084 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:52.524679 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:52.527488 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:52.528435 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:52.528571 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:52.530724 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:52.530735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:52.530740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:52.530775 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:52.530840 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:52.530872 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:52.531008 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:52.531412 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 11:11:52.531493 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:11:52.533069 29509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260925 11:11:52.533638 29509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:52.533764 29509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0919132e6b5c4eea934c0facf88a2d06, State: Initialized, Role: FOLLOWER
I20260925 11:11:52.534613 29509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260925 11:11:52.534942 29509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:52.535095 29509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:52.535312 29509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:52.539453 29509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260925 11:11:52.540189 29509 leader_election.cc:302] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0919132e6b5c4eea934c0facf88a2d06; no voters:
I20260925 11:11:52.542805 29509 leader_election.cc:288] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:52.542987 29519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:52.544152 29519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [term 1 LEADER]: Becoming Leader. State: Replica: 0919132e6b5c4eea934c0facf88a2d06, State: Running, Role: LEADER
I20260925 11:11:52.544858 29519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } }
I20260925 11:11:52.545821 29509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:52.548208 29520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0919132e6b5c4eea934c0facf88a2d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } } }
I20260925 11:11:52.548663 29520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:52.549182 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0919132e6b5c4eea934c0facf88a2d06. Latest consensus state: current_term: 1 leader_uuid: "0919132e6b5c4eea934c0facf88a2d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0919132e6b5c4eea934c0facf88a2d06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42869 } } }
I20260925 11:11:52.549491 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:52.552027 29527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:52.560676 29527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:52.569955 29527 catalog_manager.cc:1347] Generated new cluster ID: 6f52916f5ec546af8872254a02d83455
I20260925 11:11:52.570109 29527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:52.594410 29527 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:52.595883 29527 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:52.610340 29527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0919132e6b5c4eea934c0facf88a2d06: Generated new TSK 0
I20260925 11:11:52.611330 29527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:52.620221 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29442
W20260925 11:11:52.621665 29504 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51083
W20260925 11:11:52.622615 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29442: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:11:52.622825 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29442
W20260925 11:11:52.641413 29427 connection.cc:535] client connection to 127.0.0.1:42869 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42869): Connection reset by peer (error 104)
2026-09-25T11:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:52 dist-test-slave-6042 krb5kdc[29541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:52 dist-test-slave-6042 krb5kdc[29541](info): set up 2 sockets
Sep 25 11:11:52 dist-test-slave-6042 krb5kdc[29541](info): commencing operation
krb5kdc: starting...
W20260925 11:11:53.824332 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:54 dist-test-slave-6042 krb5kdc[29541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:11:54.106735 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:54.447698 29554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:54.448103 29554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:54.448282 29554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:54.455945 29554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:54.456140 29554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:54.456259 29554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:54.456369 29554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:54.456457 29554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:54.466832 29554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:54.468326 29554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:54.469749 29554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:54.482841 29560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:54.482910 29562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:54.483093 29559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:54.483952 29554 server_base.cc:1034] running on GCE node
I20260925 11:11:54.484949 29554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:54.486768 29554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:54.488165 29554 hybrid_clock.cc:648] HybridClock initialized: now 1790334714488065 us; error 68 us; skew 500 ppm
I20260925 11:11:54.488696 29554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:54 dist-test-slave-6042 krb5kdc[29541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:54.492686 29554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:11:54.494611 29554 webserver.cc:460] Webserver started at http://127.0.0.1:39591/ using document root <none> and password file <none>
I20260925 11:11:54.495483 29554 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:54.495666 29554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:54.496033 29554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:54.499711 29554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "9d8bd582f8fe4b99955927ae26f2335d"
format_stamp: "Formatted at 2026-09-25 11:11:54 on dist-test-slave-6042"
server_key: "6abed55ccc81f6dc308233a2e71e80b6"
server_key_iv: "fe9345311ca7e1f62946d33a5dddf550"
server_key_version: "encryptionkey@0"
I20260925 11:11:54.500654 29554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "9d8bd582f8fe4b99955927ae26f2335d"
format_stamp: "Formatted at 2026-09-25 11:11:54 on dist-test-slave-6042"
server_key: "6abed55ccc81f6dc308233a2e71e80b6"
server_key_iv: "fe9345311ca7e1f62946d33a5dddf550"
server_key_version: "encryptionkey@0"
I20260925 11:11:54.506937 29554 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 11:11:54.511458 29569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:54.512980 29554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 11:11:54.513226 29554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "9d8bd582f8fe4b99955927ae26f2335d"
format_stamp: "Formatted at 2026-09-25 11:11:54 on dist-test-slave-6042"
server_key: "6abed55ccc81f6dc308233a2e71e80b6"
server_key_iv: "fe9345311ca7e1f62946d33a5dddf550"
server_key_version: "encryptionkey@0"
I20260925 11:11:54.513474 29554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:54.520094 29554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:54.526254 29554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:54.526703 29554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:54.552158 29554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36435
I20260925 11:11:54.552192 29620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36435 every 8 connection(s)
I20260925 11:11:54.553843 29554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:54.557971 29621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:54.564057 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29554
I20260925 11:11:54.564247 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:54.564782 8916 external_mini_cluster.cc:1428] Setting key 4094ff76e6abdcf61aa81988cd34aa9c
I20260925 11:11:54.570600 29621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d: Bootstrap starting.
I20260925 11:11:54.575467 29621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:54.576954 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:54.567386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57601 (local address 127.0.0.1:36435)
0925 11:11:54.568611 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:54.568629 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:54.569144 (+ 515us) server_negotiation.cc:407] Connection header received
0925 11:11:54.571846 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:54.571878 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:54.572376 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:54.573023 (+ 647us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:54.574732 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:54.574741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:54.574758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:54.574780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:54.574865 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:54.574894 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:54.575106 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:54.575311 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 11:11:54.575581 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":544,"threads_started":1}
I20260925 11:11:54.576990 29621 log.cc:824] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:54.580621 29621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d: No bootstrap required, opened a new log
I20260925 11:11:54.581809 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:11:54.589334 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:54.586732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44277 (local address 127.0.0.1:36435)
0925 11:11:54.586917 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:54.586924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:54.587263 (+ 339us) server_negotiation.cc:407] Connection header received
0925 11:11:54.587409 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:54.587415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:54.587491 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:54.587617 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:54.588684 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:54.588690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:54.588694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:54.588713 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:54.588762 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:54.588782 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:54.588918 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:54.588967 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 11:11:54.589027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 11:11:54.589334 29621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260925 11:11:54.589929 29621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:54.590059 29621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8bd582f8fe4b99955927ae26f2335d, State: Initialized, Role: FOLLOWER
I20260925 11:11:54.590833 29621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260925 11:11:54.591127 29621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:54.591271 29621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:54.591449 29621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:54.593684 29621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260925 11:11:54.594204 29621 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8bd582f8fe4b99955927ae26f2335d; no voters:
I20260925 11:11:54.594888 29621 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:54.595106 29631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:54.596359 29631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [term 1 LEADER]: Becoming Leader. State: Replica: 9d8bd582f8fe4b99955927ae26f2335d, State: Running, Role: LEADER
I20260925 11:11:54.596975 29631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } }
I20260925 11:11:54.597295 29621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:54.599615 29633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8bd582f8fe4b99955927ae26f2335d. Latest consensus state: current_term: 1 leader_uuid: "9d8bd582f8fe4b99955927ae26f2335d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260925 11:11:54.600014 29633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:54.600610 29632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8bd582f8fe4b99955927ae26f2335d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8bd582f8fe4b99955927ae26f2335d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36435 } } }
I20260925 11:11:54.600909 29632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:54.601061 29638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:54.608497 29638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:54.619992 29638 catalog_manager.cc:1347] Generated new cluster ID: f7c4b683ecd4439f8d873cb2f89c29c8
I20260925 11:11:54.620162 29638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:54.656916 29638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:54.657722 29638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:54.666766 29638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8bd582f8fe4b99955927ae26f2335d: Generated new TSK 0
I20260925 11:11:54.668128 29638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:54.679880 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29554
W20260925 11:11:54.681447 29616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57601
W20260925 11:11:54.682343 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29554: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:11:54.682574 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29554
2026-09-25T11:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:54 dist-test-slave-6042 krb5kdc[29652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:54 dist-test-slave-6042 krb5kdc[29652](info): set up 2 sockets
Sep 25 11:11:54 dist-test-slave-6042 krb5kdc[29652](info): commencing operation
krb5kdc: starting...
W20260925 11:11:55.827390 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.009s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:11:56.111138 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:56.448237 29665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:56.448555 29665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:56.448725 29665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:56.456521 29665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:56.456643 29665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:56.456728 29665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:56.456837 29665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:56.456920 29665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:56.468425 29665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:56.469928 29665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:56.471659 29665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:56.485000 29673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:56.485052 29671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:56.485069 29670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:56.486207 29665 server_base.cc:1034] running on GCE node
I20260925 11:11:56.487229 29665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:56.488781 29665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:56.490021 29665 hybrid_clock.cc:648] HybridClock initialized: now 1790334716489978 us; error 41 us; skew 500 ppm
I20260925 11:11:56.490561 29665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:56.494153 29665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:11:56.495888 29665 webserver.cc:460] Webserver started at http://127.0.0.1:42471/ using document root <none> and password file <none>
I20260925 11:11:56.496670 29665 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:56.496836 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:56.497156 29665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:56.500394 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "1dbccf1df3c147e2806cc33f9c597c5b"
format_stamp: "Formatted at 2026-09-25 11:11:56 on dist-test-slave-6042"
server_key: "81800328b6e6814bbfcfb25c0d1c2830"
server_key_iv: "766143f8518f4183f20e5614ab2a9ab6"
server_key_version: "encryptionkey@0"
I20260925 11:11:56.501222 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "1dbccf1df3c147e2806cc33f9c597c5b"
format_stamp: "Formatted at 2026-09-25 11:11:56 on dist-test-slave-6042"
server_key: "81800328b6e6814bbfcfb25c0d1c2830"
server_key_iv: "766143f8518f4183f20e5614ab2a9ab6"
server_key_version: "encryptionkey@0"
I20260925 11:11:56.506722 29665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 11:11:56.510124 29680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:56.511543 29665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:11:56.511744 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "1dbccf1df3c147e2806cc33f9c597c5b"
format_stamp: "Formatted at 2026-09-25 11:11:56 on dist-test-slave-6042"
server_key: "81800328b6e6814bbfcfb25c0d1c2830"
server_key_iv: "766143f8518f4183f20e5614ab2a9ab6"
server_key_version: "encryptionkey@0"
I20260925 11:11:56.512027 29665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:56.530650 29665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:56.535393 29665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:56.535704 29665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:56.555706 29665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37641
I20260925 11:11:56.555732 29731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37641 every 8 connection(s)
I20260925 11:11:56.557143 29665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:56.558203 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29665
I20260925 11:11:56.558384 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:56.558905 8916 external_mini_cluster.cc:1428] Setting key abaa29029cccab6195e598762736021a
I20260925 11:11:56.561690 29732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 11:11:56.573225 29732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b: Bootstrap starting.
I20260925 11:11:56.576879 29732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:56.577986 29732 log.cc:824] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:56.581007 29732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b: No bootstrap required, opened a new log
I20260925 11:11:56.582391 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:56.560780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50111 (local address 127.0.0.1:37641)
0925 11:11:56.561739 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:56.561776 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:56.561832 (+ 56us) server_negotiation.cc:407] Connection header received
0925 11:11:56.563562 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:56.563584 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:56.564084 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:56.564513 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:56.566086 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:56.567503 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:56.568789 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:56.569491 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:56.569703 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:56.574387 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:56.574396 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:56.574411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:56.574524 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:56.577832 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:56.578892 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:56.578900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:56.578906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:56.579080 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:56.579585 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:56.579590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:56.579595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:56.580146 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:56.580474 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:56.580936 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 11:11:56.581368 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":455,"threads_started":1}
I20260925 11:11:56.586501 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:11:56.589818 29732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260925 11:11:56.590241 29732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:56.590350 29732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbccf1df3c147e2806cc33f9c597c5b, State: Initialized, Role: FOLLOWER
I20260925 11:11:56.591197 29732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260925 11:11:56.591521 29732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:56.591665 29732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:56.591861 29732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:56.595108 29732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260925 11:11:56.595783 29732 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbccf1df3c147e2806cc33f9c597c5b; no voters:
I20260925 11:11:56.596504 29732 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:56.596648 29742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:56.598183 29742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [term 1 LEADER]: Becoming Leader. State: Replica: 1dbccf1df3c147e2806cc33f9c597c5b, State: Running, Role: LEADER
I20260925 11:11:56.598883 29742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260925 11:11:56.599086 29732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:56.601691 29744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbccf1df3c147e2806cc33f9c597c5b. Latest consensus state: current_term: 1 leader_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } } }
I20260925 11:11:56.602135 29744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:56.602679 29743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbccf1df3c147e2806cc33f9c597c5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } } }
I20260925 11:11:56.603379 29743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:56.604126 29747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:56.611613 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:56.590981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48937 (local address 127.0.0.1:37641)
0925 11:11:56.591219 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:56.591227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:56.591730 (+ 503us) server_negotiation.cc:407] Connection header received
0925 11:11:56.591925 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:56.591929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:56.592027 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:56.592119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:56.594229 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:56.595100 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:56.596952 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:56.597665 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:56.597764 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:56.601531 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:56.601541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:56.601548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:11:56.601599 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:56.606352 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:56.607829 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:56.607837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:56.607844 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:56.608177 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:11:56.608771 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:11:56.608779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:11:56.608785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:11:56.609320 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:56.610803 (+ 1483us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:56.610970 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 11:11:56.611219 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:11:56.614078 29747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:56.628723 29747 catalog_manager.cc:1347] Generated new cluster ID: e974940eba914eb4ab263a2c85cfb239
I20260925 11:11:56.628965 29747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:56.644235 29747 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:56.646212 29747 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:56.655220 29747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbccf1df3c147e2806cc33f9c597c5b: Generated new TSK 0
I20260925 11:11:56.656514 29747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:56.674144 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29665
I20260925 11:11:56.675801 29727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:11:56.769062 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29665
2026-09-25T11:11:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29764](info): set up 2 sockets
Sep 25 11:11:56 dist-test-slave-6042 krb5kdc[29764](info): commencing operation
krb5kdc: starting...
W20260925 11:11:57.887297 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:11:58 dist-test-slave-6042 krb5kdc[29764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:11:58.167495 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:11:58.511535 29777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:11:58.511937 29777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:11:58.512118 29777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:11:58.519732 29777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:11:58.519912 29777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:11:58.520030 29777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:11:58.520126 29777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:11:58.520193 29777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:11:58.532135 29777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:11:58.534070 29777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:11:58.535609 29777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:11:58.548746 29783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:58.548834 29782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:11:58.548780 29785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:11:58.549834 29777 server_base.cc:1034] running on GCE node
I20260925 11:11:58.550822 29777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:11:58.552382 29777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:11:58.553655 29777 hybrid_clock.cc:648] HybridClock initialized: now 1790334718553624 us; error 54 us; skew 500 ppm
I20260925 11:11:58.554049 29777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:11:58 dist-test-slave-6042 krb5kdc[29764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:11:58.566246 29777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:11:58.568403 29777 webserver.cc:460] Webserver started at http://127.0.0.1:41873/ using document root <none> and password file <none>
I20260925 11:11:58.569286 29777 fs_manager.cc:359] Metadata directory not provided
I20260925 11:11:58.569464 29777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:11:58.569831 29777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:11:58.573429 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "48507c74332c4a8bb0893db43f3b7f33"
format_stamp: "Formatted at 2026-09-25 11:11:58 on dist-test-slave-6042"
server_key: "6c72b061969bb9f1dc8622e22394332b"
server_key_iv: "1e717e1a9f44b277ea880b57fe8d2f77"
server_key_version: "encryptionkey@0"
I20260925 11:11:58.574340 29777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "48507c74332c4a8bb0893db43f3b7f33"
format_stamp: "Formatted at 2026-09-25 11:11:58 on dist-test-slave-6042"
server_key: "6c72b061969bb9f1dc8622e22394332b"
server_key_iv: "1e717e1a9f44b277ea880b57fe8d2f77"
server_key_version: "encryptionkey@0"
I20260925 11:11:58.580871 29777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:11:58.585369 29792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:11:58.587301 29777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:11:58.587517 29777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "48507c74332c4a8bb0893db43f3b7f33"
format_stamp: "Formatted at 2026-09-25 11:11:58 on dist-test-slave-6042"
server_key: "6c72b061969bb9f1dc8622e22394332b"
server_key_iv: "1e717e1a9f44b277ea880b57fe8d2f77"
server_key_version: "encryptionkey@0"
I20260925 11:11:58.587764 29777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:11:58.599931 29777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:11:58.604820 29777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:11:58.605140 29777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:11:58.627035 29777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40763
I20260925 11:11:58.627049 29843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40763 every 8 connection(s)
I20260925 11:11:58.628647 29777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:11:58.632886 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:11:58.635198 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29777
I20260925 11:11:58.635387 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:11:58.635951 8916 external_mini_cluster.cc:1428] Setting key 46589a4bbcb193dbf6ac08c809be1901
I20260925 11:11:58.646610 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: Bootstrap starting.
I20260925 11:11:58.650874 29844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: Neither blocks nor log segments found. Creating new log.
I20260925 11:11:58.652348 29844 log.cc:824] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: Log is configured to *not* fsync() on all Append() calls
I20260925 11:11:58.653062 29847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:58.638283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39935 (local address 127.0.0.1:40763)
0925 11:11:58.639224 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:58.639240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:58.640900 (+ 1660us) server_negotiation.cc:407] Connection header received
0925 11:11:58.642913 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:58.642938 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:58.643409 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:58.643864 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:58.644481 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:58.645843 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:58.647419 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:58.648310 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:58.648576 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:58.650164 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:58.650173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:58.650191 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:58.650869 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:58.650977 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:58.651007 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:58.651178 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:58.651438 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 11:11:58.651804 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260925 11:11:58.656056 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: No bootstrap required, opened a new log
I20260925 11:11:58.657477 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:11:58.667066 29844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } }
I20260925 11:11:58.667542 29844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:11:58.667697 29844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48507c74332c4a8bb0893db43f3b7f33, State: Initialized, Role: FOLLOWER
I20260925 11:11:58.668409 29844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } }
I20260925 11:11:58.668721 29844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:11:58.668876 29844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:11:58.669059 29844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:11:58.670394 29847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:11:58.663343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59333 (local address 127.0.0.1:40763)
0925 11:11:58.663595 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 11:11:58.663602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:11:58.663881 (+ 279us) server_negotiation.cc:407] Connection header received
0925 11:11:58.664066 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:11:58.664072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:11:58.664155 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:11:58.664284 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:11:58.664800 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:58.665976 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:58.667568 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:11:58.668128 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:11:58.668281 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:11:58.669599 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:11:58.669605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:11:58.669608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:11:58.669630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:11:58.669674 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:11:58.669691 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:11:58.669807 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 11:11:58.669972 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 11:11:58.670076 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 11:11:58.672202 29844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } }
I20260925 11:11:58.672771 29844 leader_election.cc:302] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48507c74332c4a8bb0893db43f3b7f33; no voters:
I20260925 11:11:58.673453 29844 leader_election.cc:288] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:11:58.673520 29854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:11:58.675781 29844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:11:58.678833 29854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [term 1 LEADER]: Becoming Leader. State: Replica: 48507c74332c4a8bb0893db43f3b7f33, State: Running, Role: LEADER
I20260925 11:11:58.679553 29854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } }
I20260925 11:11:58.683624 29855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48507c74332c4a8bb0893db43f3b7f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } } }
I20260925 11:11:58.684305 29855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [sys.catalog]: This master's current role is: LEADER
I20260925 11:11:58.685118 29856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48507c74332c4a8bb0893db43f3b7f33. Latest consensus state: current_term: 1 leader_uuid: "48507c74332c4a8bb0893db43f3b7f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48507c74332c4a8bb0893db43f3b7f33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40763 } } }
I20260925 11:11:58.685464 29856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33 [sys.catalog]: This master's current role is: LEADER
W20260925 11:11:58.689639 29868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:11:58.689801 29868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:11:58.689960 29869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:11:58.691993 29869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:11:58.701232 29869 catalog_manager.cc:1347] Generated new cluster ID: e8ec80c3515c40bfbc2b71a0ed2c0380
I20260925 11:11:58.701422 29869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:11:58.741146 29869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:11:58.743036 29869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:11:58.758347 29869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48507c74332c4a8bb0893db43f3b7f33: Generated new TSK 0
I20260925 11:11:58.759424 29869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:11:58.768024 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29777
W20260925 11:11:58.769321 29839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39935
W20260925 11:11:58.770323 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:11:58.770540 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29777
2026-09-25T11:11:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:11:58 dist-test-slave-6042 krb5kdc[29875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:11:58 dist-test-slave-6042 krb5kdc[29875](info): set up 2 sockets
Sep 25 11:11:58 dist-test-slave-6042 krb5kdc[29875](info): commencing operation
krb5kdc: starting...
W20260925 11:11:59.899307 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:00 dist-test-slave-6042 krb5kdc[29875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:00.178891 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:00.517624 29888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:00.518044 29888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:00.518208 29888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:00.526124 29888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:00.526300 29888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:00.526392 29888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:00.526526 29888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:00.526598 29888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:00.539155 29888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:00.540575 29888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:00.541950 29888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:00.555207 29893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:00.555202 29894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:00.555146 29896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:00.555810 29888 server_base.cc:1034] running on GCE node
I20260925 11:12:00.556689 29888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:00.558197 29888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:00.559475 29888 hybrid_clock.cc:648] HybridClock initialized: now 1790334720559426 us; error 50 us; skew 500 ppm
I20260925 11:12:00.559890 29888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:00 dist-test-slave-6042 krb5kdc[29875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:00.563413 29888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:00.565181 29888 webserver.cc:460] Webserver started at http://127.0.0.1:43039/ using document root <none> and password file <none>
I20260925 11:12:00.565920 29888 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:00.566082 29888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:00.566399 29888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:00.569537 29888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "98c30726fbb54fec8e8c97b22844143a"
format_stamp: "Formatted at 2026-09-25 11:12:00 on dist-test-slave-6042"
server_key: "28fb5cdece1b3aa77625e78849536abb"
server_key_iv: "598c05fa0589b0c79fbb6adc59749913"
server_key_version: "encryptionkey@0"
I20260925 11:12:00.570355 29888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "98c30726fbb54fec8e8c97b22844143a"
format_stamp: "Formatted at 2026-09-25 11:12:00 on dist-test-slave-6042"
server_key: "28fb5cdece1b3aa77625e78849536abb"
server_key_iv: "598c05fa0589b0c79fbb6adc59749913"
server_key_version: "encryptionkey@0"
I20260925 11:12:00.576086 29888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 11:12:00.580121 29903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:00.581720 29888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:12:00.581938 29888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "98c30726fbb54fec8e8c97b22844143a"
format_stamp: "Formatted at 2026-09-25 11:12:00 on dist-test-slave-6042"
server_key: "28fb5cdece1b3aa77625e78849536abb"
server_key_iv: "598c05fa0589b0c79fbb6adc59749913"
server_key_version: "encryptionkey@0"
I20260925 11:12:00.582182 29888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:00.588505 29888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:00.593632 29888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:00.594029 29888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:00.614965 29888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44661
I20260925 11:12:00.615015 29954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44661 every 8 connection(s)
I20260925 11:12:00.616632 29888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:00.620777 29955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:00.626391 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29888
I20260925 11:12:00.626649 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:00.627210 8916 external_mini_cluster.cc:1428] Setting key 02d176f4e431108d5c0fcda263794091
I20260925 11:12:00.633611 29955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a: Bootstrap starting.
I20260925 11:12:00.635876 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:00.629274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57051 (local address 127.0.0.1:44661)
0925 11:12:00.630225 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:00.630239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:00.630282 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:12:00.632249 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:00.632275 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:00.632669 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:00.633085 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:00.634223 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:00.634230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:00.634243 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:00.634255 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:00.634309 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:00.634328 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:00.634599 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:00.634737 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 11:12:00.634936 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":409,"threads_started":1}
I20260925 11:12:00.637190 29955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:00.638911 29955 log.cc:824] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:00.640568 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:00.642124 29955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a: No bootstrap required, opened a new log
I20260925 11:12:00.647997 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:00.645247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45407 (local address 127.0.0.1:44661)
0925 11:12:00.645455 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:00.645466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:00.645860 (+ 394us) server_negotiation.cc:407] Connection header received
0925 11:12:00.646009 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:00.646014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:00.646075 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:00.646184 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:00.647238 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:00.647245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:00.647250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:00.647269 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:00.647323 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:00.647343 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:00.647485 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:00.647621 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 11:12:00.647688 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 11:12:00.650900 29955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260925 11:12:00.651320 29955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:00.651396 29955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c30726fbb54fec8e8c97b22844143a, State: Initialized, Role: FOLLOWER
I20260925 11:12:00.651927 29955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260925 11:12:00.652202 29955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:00.652380 29955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:00.652575 29955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:00.655047 29955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260925 11:12:00.655649 29955 leader_election.cc:302] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c30726fbb54fec8e8c97b22844143a; no voters:
I20260925 11:12:00.656366 29955 leader_election.cc:288] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:00.656512 29965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:00.657256 29965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [term 1 LEADER]: Becoming Leader. State: Replica: 98c30726fbb54fec8e8c97b22844143a, State: Running, Role: LEADER
I20260925 11:12:00.657810 29965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } }
I20260925 11:12:00.658730 29955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:00.660895 29966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c30726fbb54fec8e8c97b22844143a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260925 11:12:00.661397 29966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:00.663483 29972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:00.663401 29967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c30726fbb54fec8e8c97b22844143a. Latest consensus state: current_term: 1 leader_uuid: "98c30726fbb54fec8e8c97b22844143a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c30726fbb54fec8e8c97b22844143a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44661 } } }
I20260925 11:12:00.663798 29967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:00.669299 29972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:00.679984 29972 catalog_manager.cc:1347] Generated new cluster ID: a03b36a0aa5843fb861f3f94230fe967
I20260925 11:12:00.680160 29972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:00.694021 29972 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:00.694741 29972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:00.703392 29972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c30726fbb54fec8e8c97b22844143a: Generated new TSK 0
I20260925 11:12:00.704468 29972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:00.718707 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29888
W20260925 11:12:00.720191 29950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57051
W20260925 11:12:00.720955 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:00.721091 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29888
2026-09-25T11:12:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:00 dist-test-slave-6042 krb5kdc[29986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:00 dist-test-slave-6042 krb5kdc[29986](info): set up 2 sockets
Sep 25 11:12:00 dist-test-slave-6042 krb5kdc[29986](info): commencing operation
krb5kdc: starting...
W20260925 11:12:01.867931 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[29986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:02.150559 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:02.495734 29999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:02.496091 29999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:02.496186 29999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:02.503850 29999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:02.503993 29999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:02.504040 29999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:02.504096 29999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:02.504141 29999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:02.514607 29999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:02.515990 29999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:02.517347 29999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:02.533650 30005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:02.533656 30004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:02.535475 30007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:02.535454 29999 server_base.cc:1034] running on GCE node
I20260925 11:12:02.536485 29999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:02.538120 29999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:02.539441 29999 hybrid_clock.cc:648] HybridClock initialized: now 1790334722539388 us; error 50 us; skew 500 ppm
I20260925 11:12:02.539884 29999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[29986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:02.543596 29999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:02.545388 29999 webserver.cc:460] Webserver started at http://127.0.0.1:40643/ using document root <none> and password file <none>
I20260925 11:12:02.546166 29999 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:02.546325 29999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:02.546742 29999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:02.549973 29999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "e6fc731a794c415abd216531100b1932"
format_stamp: "Formatted at 2026-09-25 11:12:02 on dist-test-slave-6042"
server_key: "02ead541c18b04897cbcf27e9a473a19"
server_key_iv: "75f8fd826079f5467d718d8ffb1b2b78"
server_key_version: "encryptionkey@0"
I20260925 11:12:02.550940 29999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "e6fc731a794c415abd216531100b1932"
format_stamp: "Formatted at 2026-09-25 11:12:02 on dist-test-slave-6042"
server_key: "02ead541c18b04897cbcf27e9a473a19"
server_key_iv: "75f8fd826079f5467d718d8ffb1b2b78"
server_key_version: "encryptionkey@0"
I20260925 11:12:02.556918 29999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 11:12:02.561161 30014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:02.563264 29999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 11:12:02.563474 29999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "e6fc731a794c415abd216531100b1932"
format_stamp: "Formatted at 2026-09-25 11:12:02 on dist-test-slave-6042"
server_key: "02ead541c18b04897cbcf27e9a473a19"
server_key_iv: "75f8fd826079f5467d718d8ffb1b2b78"
server_key_version: "encryptionkey@0"
I20260925 11:12:02.563791 29999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:02.584299 29999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:02.589659 29999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:02.590010 29999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:02.611090 29999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34259
I20260925 11:12:02.611127 30065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34259 every 8 connection(s)
I20260925 11:12:02.612603 29999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:02.617406 30066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:02.618836 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 29999
I20260925 11:12:02.619046 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:02.619612 8916 external_mini_cluster.cc:1428] Setting key 28c0ff6beba12ea35696d854b06d1033
I20260925 11:12:02.633245 30066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932: Bootstrap starting.
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[29986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 11:12:02.637328 30066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:02.638549 30066 log.cc:824] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:02.641639 30066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932: No bootstrap required, opened a new log
I20260925 11:12:02.645414 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:02.621719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:45953 (local address 127.0.0.1:34259)
0925 11:12:02.622717 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:02.622735 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:02.622784 (+ 49us) server_negotiation.cc:407] Connection header received
0925 11:12:02.624678 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:02.624708 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:02.625280 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:02.625702 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:02.627702 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:02.629057 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:02.631169 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:02.631967 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:02.632139 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:02.637314 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:02.637324 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:02.637341 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:02.637385 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:02.640856 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:02.641808 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:02.641814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:02.641821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:02.641990 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:02.642530 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:02.642534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:02.642536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:02.643091 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:02.643479 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:02.643991 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 11:12:02.644330 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":443,"threads_started":1}
I20260925 11:12:02.649693 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:02.650187 30066 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260925 11:12:02.650594 30066 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:02.650729 30066 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fc731a794c415abd216531100b1932, State: Initialized, Role: FOLLOWER
I20260925 11:12:02.651312 30066 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260925 11:12:02.651553 30066 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:02.651701 30066 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:02.651877 30066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:02.654332 30066 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260925 11:12:02.655000 30066 leader_election.cc:302] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6fc731a794c415abd216531100b1932; no voters:
I20260925 11:12:02.655789 30066 leader_election.cc:288] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:02.655926 30076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:02.656780 30076 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [term 1 LEADER]: Becoming Leader. State: Replica: e6fc731a794c415abd216531100b1932, State: Running, Role: LEADER
I20260925 11:12:02.657413 30076 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260925 11:12:02.659750 30066 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:02.660364 30078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6fc731a794c415abd216531100b1932. Latest consensus state: current_term: 1 leader_uuid: "e6fc731a794c415abd216531100b1932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260925 11:12:02.660779 30078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:02.665750 30077 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6fc731a794c415abd216531100b1932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6fc731a794c415abd216531100b1932" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260925 11:12:02.666884 30077 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:02.672458 30080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:02.675724 30069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:02.654789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:54457 (local address 127.0.0.1:34259)
0925 11:12:02.655054 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:02.655061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:02.655418 (+ 357us) server_negotiation.cc:407] Connection header received
0925 11:12:02.655635 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:02.655643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:02.655746 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:02.655873 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:02.657967 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:02.658960 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:02.660955 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:02.661640 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:02.661759 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:02.666430 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:02.666441 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:02.666449 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:02.666555 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:02.671564 (+ 5009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:02.672939 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:02.672950 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:02.672953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:02.673150 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:02.673930 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:02.673935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:02.673939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:02.674344 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:02.674749 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:02.675145 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 11:12:02.675336 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:12:02.683310 30080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:02.695981 30080 catalog_manager.cc:1347] Generated new cluster ID: 207aa35a7ea04b61ac0da33f77a3b1f4
I20260925 11:12:02.696121 30080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:02.714404 30080 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:02.716427 30080 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:02.724743 30080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6fc731a794c415abd216531100b1932: Generated new TSK 0
I20260925 11:12:02.725622 30080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:02.736228 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 29999
I20260925 11:12:02.737380 30061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:02.833256 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 29999
W20260925 11:12:02.851708 29984 connection.cc:535] client connection to 127.0.0.1:34259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34259): Connection reset by peer (error 104)
2026-09-25T11:12:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2112 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[30098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[30098](info): set up 2 sockets
Sep 25 11:12:02 dist-test-slave-6042 krb5kdc[30098](info): commencing operation
krb5kdc: starting...
W20260925 11:12:03.959678 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:04 dist-test-slave-6042 krb5kdc[30098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:04.245995 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:04.599702 30111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:04.600041 30111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:04.600199 30111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:04.608741 30111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:04.608886 30111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:04.609205 30111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:04.609371 30111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:04.609438 30111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:04.619962 30111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:04.621402 30111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:04.622901 30111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:04.635665 30117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:04.635782 30116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:04.635648 30119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:04.636579 30111 server_base.cc:1034] running on GCE node
I20260925 11:12:04.637537 30111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:04.639274 30111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:04.640535 30111 hybrid_clock.cc:648] HybridClock initialized: now 1790334724640483 us; error 51 us; skew 500 ppm
I20260925 11:12:04.640949 30111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:04 dist-test-slave-6042 krb5kdc[30098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:04.650926 30111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:04.652871 30111 webserver.cc:460] Webserver started at http://127.0.0.1:35297/ using document root <none> and password file <none>
I20260925 11:12:04.653682 30111 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:04.653854 30111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:04.654201 30111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:04.657599 30111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "9815a71f09364c8bbfdb7b53a5daf352"
format_stamp: "Formatted at 2026-09-25 11:12:04 on dist-test-slave-6042"
server_key: "0db96aef403c403e4b830f729ac3c28a"
server_key_iv: "c9723ba590b29700c33b247f7b0f690a"
server_key_version: "encryptionkey@0"
I20260925 11:12:04.658435 30111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "9815a71f09364c8bbfdb7b53a5daf352"
format_stamp: "Formatted at 2026-09-25 11:12:04 on dist-test-slave-6042"
server_key: "0db96aef403c403e4b830f729ac3c28a"
server_key_iv: "c9723ba590b29700c33b247f7b0f690a"
server_key_version: "encryptionkey@0"
I20260925 11:12:04.663978 30111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:12:04.667778 30126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:04.669205 30111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:12:04.669404 30111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "9815a71f09364c8bbfdb7b53a5daf352"
format_stamp: "Formatted at 2026-09-25 11:12:04 on dist-test-slave-6042"
server_key: "0db96aef403c403e4b830f729ac3c28a"
server_key_iv: "c9723ba590b29700c33b247f7b0f690a"
server_key_version: "encryptionkey@0"
I20260925 11:12:04.669623 30111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:04.685344 30111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:04.690052 30111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:04.690368 30111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:04.710829 30111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40581
I20260925 11:12:04.710853 30177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40581 every 8 connection(s)
I20260925 11:12:04.713267 30111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:04.714877 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30111
I20260925 11:12:04.715060 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:04.715651 8916 external_mini_cluster.cc:1428] Setting key 279340c56a166a1461a92558b0e9e8a0
I20260925 11:12:04.721501 30178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:04.739537 30178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352: Bootstrap starting.
I20260925 11:12:04.744046 30178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:04.745607 30178 log.cc:824] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:04.746295 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:04.717922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:46671 (local address 127.0.0.1:40581)
0925 11:12:04.731092 (+ 13170us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:04.731110 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:04.731172 (+ 62us) server_negotiation.cc:407] Connection header received
0925 11:12:04.733798 (+ 2626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:04.733836 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:04.734588 (+ 752us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:04.735098 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:04.736588 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:04.738724 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:04.740649 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:04.741391 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:04.741534 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:04.743070 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:04.743083 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:04.743114 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:04.743836 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:04.743956 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:04.743993 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:04.744212 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:04.744574 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 11:12:04.744801 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12867,"thread_start_us":12633,"threads_started":1}
I20260925 11:12:04.749826 30178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352: No bootstrap required, opened a new log
I20260925 11:12:04.750556 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:04.764303 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:04.756559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:58631 (local address 127.0.0.1:40581)
0925 11:12:04.756814 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:04.756828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:04.757426 (+ 598us) server_negotiation.cc:407] Connection header received
0925 11:12:04.757662 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:04.757669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:04.757762 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:04.757854 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:04.758694 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:04.759515 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:04.761573 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:04.762038 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:04.762199 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:04.763500 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:04.763505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:04.763509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:04.763528 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:04.763568 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:04.763584 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:04.763708 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:04.763889 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 11:12:04.763947 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 11:12:04.763898 30178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260925 11:12:04.764503 30178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:04.764645 30178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9815a71f09364c8bbfdb7b53a5daf352, State: Initialized, Role: FOLLOWER
I20260925 11:12:04.765388 30178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260925 11:12:04.765664 30178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:04.765805 30178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:04.765995 30178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:04.768286 30178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260925 11:12:04.768821 30178 leader_election.cc:302] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9815a71f09364c8bbfdb7b53a5daf352; no voters:
I20260925 11:12:04.769480 30178 leader_election.cc:288] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:04.769564 30188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:04.771090 30188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [term 1 LEADER]: Becoming Leader. State: Replica: 9815a71f09364c8bbfdb7b53a5daf352, State: Running, Role: LEADER
I20260925 11:12:04.771584 30178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:04.771734 30188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } }
I20260925 11:12:04.775117 30190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9815a71f09364c8bbfdb7b53a5daf352. Latest consensus state: current_term: 1 leader_uuid: "9815a71f09364c8bbfdb7b53a5daf352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } } }
I20260925 11:12:04.775157 30189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9815a71f09364c8bbfdb7b53a5daf352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9815a71f09364c8bbfdb7b53a5daf352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40581 } } }
I20260925 11:12:04.775622 30189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:04.775600 30190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:04.776520 30198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:04.781993 30198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:04.782754 30205 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 11:12:04.791033 30198 catalog_manager.cc:1347] Generated new cluster ID: b6e525eb5ab244b7ac4c5c729b8f1222
I20260925 11:12:04.791193 30198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:04.803490 30198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:04.804895 30198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:04.821435 30198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9815a71f09364c8bbfdb7b53a5daf352: Generated new TSK 0
I20260925 11:12:04.822530 30198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:04.829680 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30111
W20260925 11:12:04.830962 30173 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.176:46671
W20260925 11:12:04.831774 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30111: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:04.831935 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30111
W20260925 11:12:04.850441 30096 connection.cc:535] client connection to 127.0.0.1:40581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40581): Connection reset by peer (error 104)
2026-09-25T11:12:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:04 dist-test-slave-6042 krb5kdc[30209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:04 dist-test-slave-6042 krb5kdc[30209](info): set up 2 sockets
Sep 25 11:12:04 dist-test-slave-6042 krb5kdc[30209](info): commencing operation
krb5kdc: starting...
W20260925 11:12:05.999473 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:06 dist-test-slave-6042 krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:06.283874 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:06.633486 30222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:06.633809 30222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:06.634004 30222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:06.641697 30222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:06.641827 30222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:06.641906 30222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:06.641996 30222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:06.642088 30222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:06.652619 30222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:06.654050 30222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:06.655498 30222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:06.668473 30227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:06.668488 30228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:06.668718 30230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:06.669265 30222 server_base.cc:1034] running on GCE node
I20260925 11:12:06.670115 30222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:06.671891 30222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:06.673166 30222 hybrid_clock.cc:648] HybridClock initialized: now 1790334726673092 us; error 72 us; skew 500 ppm
I20260925 11:12:06.673579 30222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:06 dist-test-slave-6042 krb5kdc[30209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:06.677100 30222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:06.678772 30222 webserver.cc:460] Webserver started at http://127.0.0.1:46161/ using document root <none> and password file <none>
I20260925 11:12:06.679491 30222 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:06.679597 30222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:06.679850 30222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:06.683072 30222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "c5cc9dc478df4145b8f1ad3576a1461a"
format_stamp: "Formatted at 2026-09-25 11:12:06 on dist-test-slave-6042"
server_key: "231861cc63d0a41fe755a7fd41fc7a23"
server_key_iv: "7e05e63f5fe7768ebe3e710b6f5f3a58"
server_key_version: "encryptionkey@0"
I20260925 11:12:06.683840 30222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "c5cc9dc478df4145b8f1ad3576a1461a"
format_stamp: "Formatted at 2026-09-25 11:12:06 on dist-test-slave-6042"
server_key: "231861cc63d0a41fe755a7fd41fc7a23"
server_key_iv: "7e05e63f5fe7768ebe3e710b6f5f3a58"
server_key_version: "encryptionkey@0"
I20260925 11:12:06.689661 30222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 11:12:06.693904 30237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:06.695730 30222 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260925 11:12:06.695942 30222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "c5cc9dc478df4145b8f1ad3576a1461a"
format_stamp: "Formatted at 2026-09-25 11:12:06 on dist-test-slave-6042"
server_key: "231861cc63d0a41fe755a7fd41fc7a23"
server_key_iv: "7e05e63f5fe7768ebe3e710b6f5f3a58"
server_key_version: "encryptionkey@0"
I20260925 11:12:06.696151 30222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:06.702792 30222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:06.707401 30222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:06.707736 30222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:06.728376 30222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42459
I20260925 11:12:06.728394 30288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42459 every 8 connection(s)
I20260925 11:12:06.729854 30222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:06.733776 30289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:06.735843 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30222
I20260925 11:12:06.736109 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:06.736577 8916 external_mini_cluster.cc:1428] Setting key 09324be649fa8e35cd7f8dd76bd65009
I20260925 11:12:06.746591 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:06.738870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:37321 (local address 127.0.0.1:42459)
0925 11:12:06.739807 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:06.739825 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:06.739881 (+ 56us) server_negotiation.cc:407] Connection header received
0925 11:12:06.742347 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:06.742381 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:06.742895 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:06.743376 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:06.744433 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:06.744442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:06.744459 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:06.744482 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:06.744554 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:06.744587 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:06.744835 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:06.744983 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 11:12:06.745212 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":402,"threads_started":1}
I20260925 11:12:06.748665 30289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a: Bootstrap starting.
I20260925 11:12:06.750972 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:06.752305 30289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:06.753379 30289 log.cc:824] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:06.757035 30289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a: No bootstrap required, opened a new log
I20260925 11:12:06.758311 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:06.755416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:45309 (local address 127.0.0.1:42459)
0925 11:12:06.755646 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:06.755656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:06.756124 (+ 468us) server_negotiation.cc:407] Connection header received
0925 11:12:06.756289 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:06.756293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:06.756343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:06.756430 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:06.757635 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:06.757640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:06.757643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:06.757659 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:06.757698 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:06.757714 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:06.757841 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:06.757966 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 11:12:06.758043 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:12:06.765992 30289 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260925 11:12:06.766454 30289 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:06.766669 30289 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cc9dc478df4145b8f1ad3576a1461a, State: Initialized, Role: FOLLOWER
I20260925 11:12:06.767290 30289 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260925 11:12:06.767570 30289 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:06.767727 30289 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:06.767908 30289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:06.770678 30289 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260925 11:12:06.771278 30289 leader_election.cc:302] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5cc9dc478df4145b8f1ad3576a1461a; no voters:
I20260925 11:12:06.771943 30289 leader_election.cc:288] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:06.772176 30299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:06.773000 30299 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [term 1 LEADER]: Becoming Leader. State: Replica: c5cc9dc478df4145b8f1ad3576a1461a, State: Running, Role: LEADER
I20260925 11:12:06.773573 30299 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260925 11:12:06.774158 30289 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:06.783350 30301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5cc9dc478df4145b8f1ad3576a1461a. Latest consensus state: current_term: 1 leader_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260925 11:12:06.783730 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cc9dc478df4145b8f1ad3576a1461a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260925 11:12:06.784307 30301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:06.784446 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:06.785418 30313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:06.791281 30313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:06.803314 30313 catalog_manager.cc:1347] Generated new cluster ID: b4990ab5466c4a5387d6ca3a0613d688
I20260925 11:12:06.803467 30313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:06.827385 30313 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:06.827922 30313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:06.837141 30313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5cc9dc478df4145b8f1ad3576a1461a: Generated new TSK 0
I20260925 11:12:06.838057 30313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:06.846020 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30222
W20260925 11:12:06.847332 30284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.176:37321
W20260925 11:12:06.848085 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:06.848260 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30222
2026-09-25T11:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:06 dist-test-slave-6042 krb5kdc[30320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:06 dist-test-slave-6042 krb5kdc[30320](info): set up 2 sockets
Sep 25 11:12:06 dist-test-slave-6042 krb5kdc[30320](info): commencing operation
krb5kdc: starting...
W20260925 11:12:07.967674 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:08 dist-test-slave-6042 krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:08.260001 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:08.617589 30333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:08.618002 30333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:08.618199 30333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:08.625828 30333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:08.626016 30333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:08.626228 30333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:08.626348 30333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:08.626430 30333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:08.637084 30333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:08.638602 30333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:08.639973 30333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:08.652841 30341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:08.652935 30338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:08.652899 30339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:08.654076 30333 server_base.cc:1034] running on GCE node
I20260925 11:12:08.655107 30333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:08.656749 30333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:08.658006 30333 hybrid_clock.cc:648] HybridClock initialized: now 1790334728657966 us; error 65 us; skew 500 ppm
I20260925 11:12:08.658448 30333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:08 dist-test-slave-6042 krb5kdc[30320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:08.662020 30333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:08.663980 30333 webserver.cc:460] Webserver started at http://127.0.0.1:35061/ using document root <none> and password file <none>
I20260925 11:12:08.664770 30333 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:08.664932 30333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:08.665271 30333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:08.668582 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "62b7249d55b147b98a6e33f196f91c9e"
format_stamp: "Formatted at 2026-09-25 11:12:08 on dist-test-slave-6042"
server_key: "b68333685d70c366bab48070d840dd94"
server_key_iv: "07941e185562a502cdb01500c3135564"
server_key_version: "encryptionkey@0"
I20260925 11:12:08.669456 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "62b7249d55b147b98a6e33f196f91c9e"
format_stamp: "Formatted at 2026-09-25 11:12:08 on dist-test-slave-6042"
server_key: "b68333685d70c366bab48070d840dd94"
server_key_iv: "07941e185562a502cdb01500c3135564"
server_key_version: "encryptionkey@0"
I20260925 11:12:08.675630 30333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 11:12:08.679507 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:08.681147 30333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 11:12:08.681397 30333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "62b7249d55b147b98a6e33f196f91c9e"
format_stamp: "Formatted at 2026-09-25 11:12:08 on dist-test-slave-6042"
server_key: "b68333685d70c366bab48070d840dd94"
server_key_iv: "07941e185562a502cdb01500c3135564"
server_key_version: "encryptionkey@0"
I20260925 11:12:08.681666 30333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:08.706782 30333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:08.711997 30333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:08.712272 30333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:08.733314 30333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43449
I20260925 11:12:08.733335 30399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43449 every 8 connection(s)
I20260925 11:12:08.734962 30333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:08.739161 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:08.740367 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30333
I20260925 11:12:08.740578 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:08.741195 8916 external_mini_cluster.cc:1428] Setting key 9ca91942775ae94c909eaa5af26af7be
I20260925 11:12:08.752978 30400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e: Bootstrap starting.
Sep 25 11:12:08 dist-test-slave-6042 krb5kdc[30320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 11:12:08.756579 30400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:08.757719 30400 log.cc:824] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:08.760886 30400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e: No bootstrap required, opened a new log
I20260925 11:12:08.765154 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:08.743212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:58819 (local address 127.0.0.1:43449)
0925 11:12:08.744052 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:08.744065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:08.744186 (+ 121us) server_negotiation.cc:407] Connection header received
0925 11:12:08.745955 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:08.745982 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:08.746578 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:08.747046 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:08.748634 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:08.749994 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:08.751431 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:08.752196 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:08.752411 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:08.757251 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:08.757260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:08.757276 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:08.757322 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:08.760542 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:08.761510 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:08.761517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:08.761524 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:08.761704 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:08.762198 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:08.762201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:08.762204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:08.762829 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:08.763207 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:08.763730 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 11:12:08.764064 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":368,"threads_started":1}
I20260925 11:12:08.769622 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:08.770056 30400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260925 11:12:08.770542 30400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:08.770659 30400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b7249d55b147b98a6e33f196f91c9e, State: Initialized, Role: FOLLOWER
I20260925 11:12:08.771366 30400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260925 11:12:08.771621 30400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:08.771775 30400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:08.771965 30400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:08.774405 30400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260925 11:12:08.775101 30400 leader_election.cc:302] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b7249d55b147b98a6e33f196f91c9e; no voters:
I20260925 11:12:08.775833 30400 leader_election.cc:288] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:08.775985 30410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:08.777233 30410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [term 1 LEADER]: Becoming Leader. State: Replica: 62b7249d55b147b98a6e33f196f91c9e, State: Running, Role: LEADER
I20260925 11:12:08.777916 30410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260925 11:12:08.778391 30400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:08.781136 30411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b7249d55b147b98a6e33f196f91c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260925 11:12:08.781646 30411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:08.786156 30412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b7249d55b147b98a6e33f196f91c9e. Latest consensus state: current_term: 1 leader_uuid: "62b7249d55b147b98a6e33f196f91c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b7249d55b147b98a6e33f196f91c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260925 11:12:08.786629 30412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:08.790315 30415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:08.798749 30415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:08.799580 30403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:08.775151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:41789 (local address 127.0.0.1:43449)
0925 11:12:08.775356 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:08.775363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:08.775899 (+ 536us) server_negotiation.cc:407] Connection header received
0925 11:12:08.776131 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:08.776139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:08.776254 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:08.776394 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:08.778705 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:08.779765 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:08.782090 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:08.783077 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:08.783264 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:08.788756 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:08.788767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:08.788774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:08.788834 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:08.793280 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:08.794794 (+ 1514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:08.794808 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:08.794815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:08.795095 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:08.795972 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:08.795981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:08.795987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:08.796573 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:08.796975 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:08.798598 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0925 11:12:08.799003 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:12:08.810014 30415 catalog_manager.cc:1347] Generated new cluster ID: a1da772183dd4da5b8e14d3b936091d2
I20260925 11:12:08.810276 30415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:08.822786 30415 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:08.824303 30415 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:08.835219 30415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b7249d55b147b98a6e33f196f91c9e: Generated new TSK 0
I20260925 11:12:08.836225 30415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:08.849669 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30333
I20260925 11:12:08.851832 30395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:08.947479 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30333
W20260925 11:12:08.965343 30318 connection.cc:535] client connection to 127.0.0.1:43449 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43449): Connection reset by peer (error 104)
2026-09-25T11:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:09 dist-test-slave-6042 krb5kdc[30432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:09 dist-test-slave-6042 krb5kdc[30432](info): set up 2 sockets
Sep 25 11:12:09 dist-test-slave-6042 krb5kdc[30432](info): commencing operation
krb5kdc: starting...
W20260925 11:12:10.087359 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:10 dist-test-slave-6042 krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:10.369853 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:10.719108 30445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:10.719439 30445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:10.719640 30445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:10.728123 30445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:10.728266 30445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:10.728387 30445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:10.728510 30445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:10.728598 30445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:10.740707 30445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:10.742259 30445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:10.743769 30445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:10.757483 30450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:10.757414 30453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:10.757447 30451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:10.758466 30445 server_base.cc:1034] running on GCE node
I20260925 11:12:10.759374 30445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:10.760978 30445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:10.762229 30445 hybrid_clock.cc:648] HybridClock initialized: now 1790334730762180 us; error 50 us; skew 500 ppm
I20260925 11:12:10.762661 30445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:10 dist-test-slave-6042 krb5kdc[30432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:10.771981 30445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:10.773761 30445 webserver.cc:460] Webserver started at http://127.0.0.1:44223/ using document root <none> and password file <none>
I20260925 11:12:10.774636 30445 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:10.774808 30445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:10.775131 30445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:10.778330 30445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "a78fa660119e4a638a790d31587350a7"
format_stamp: "Formatted at 2026-09-25 11:12:10 on dist-test-slave-6042"
server_key: "8ac76bebe4a6b9b5199db41c65bd06f6"
server_key_iv: "6900696112affae848ef5b760f6ae59f"
server_key_version: "encryptionkey@0"
I20260925 11:12:10.779263 30445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "a78fa660119e4a638a790d31587350a7"
format_stamp: "Formatted at 2026-09-25 11:12:10 on dist-test-slave-6042"
server_key: "8ac76bebe4a6b9b5199db41c65bd06f6"
server_key_iv: "6900696112affae848ef5b760f6ae59f"
server_key_version: "encryptionkey@0"
I20260925 11:12:10.784927 30445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:12:10.788681 30460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:10.790222 30445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:12:10.790444 30445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "a78fa660119e4a638a790d31587350a7"
format_stamp: "Formatted at 2026-09-25 11:12:10 on dist-test-slave-6042"
server_key: "8ac76bebe4a6b9b5199db41c65bd06f6"
server_key_iv: "6900696112affae848ef5b760f6ae59f"
server_key_version: "encryptionkey@0"
I20260925 11:12:10.790748 30445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:10.816601 30445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:10.821939 30445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:10.822312 30445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:10.843168 30445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41285
I20260925 11:12:10.843206 30511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41285 every 8 connection(s)
I20260925 11:12:10.844662 30445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:10.848383 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30445
I20260925 11:12:10.848644 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:10.849114 8916 external_mini_cluster.cc:1428] Setting key a0ed41c1ce8c939f33b79e364f972cdc
I20260925 11:12:10.849291 30512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:10.866377 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7: Bootstrap starting.
I20260925 11:12:10.866640 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:10.851237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:54901 (local address 127.0.0.1:41285)
0925 11:12:10.852156 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:10.852173 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:10.852227 (+ 54us) server_negotiation.cc:407] Connection header received
0925 11:12:10.854959 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:10.854994 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:10.855660 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:10.856206 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:10.856813 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:10.859158 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:10.861203 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:10.862387 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:10.862681 (+ 294us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:10.864091 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:10.864099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:10.864116 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:10.864718 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:10.864810 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:10.864838 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:10.865006 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:10.865320 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 11:12:10.865514 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":385,"threads_started":1}
I20260925 11:12:10.870739 30512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:10.871019 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:10.872033 30512 log.cc:824] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:10.875542 30512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7: No bootstrap required, opened a new log
I20260925 11:12:10.884150 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:10.875816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:35835 (local address 127.0.0.1:41285)
0925 11:12:10.876066 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:10.876076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:10.876592 (+ 516us) server_negotiation.cc:407] Connection header received
0925 11:12:10.876812 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:10.876816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:10.876881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:10.876967 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:10.877703 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:10.878725 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:10.880911 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:10.881391 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:10.881501 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:10.883128 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:10.883133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:10.883137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:10.883158 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:10.883220 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:10.883240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:10.883334 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:10.883613 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 11:12:10.883715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:12:10.886907 30512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } }
I20260925 11:12:10.887383 30512 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:10.887521 30512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78fa660119e4a638a790d31587350a7, State: Initialized, Role: FOLLOWER
I20260925 11:12:10.888165 30512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } }
I20260925 11:12:10.888455 30512 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:10.888620 30512 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:10.888818 30512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:10.891480 30512 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } }
I20260925 11:12:10.892109 30512 leader_election.cc:302] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78fa660119e4a638a790d31587350a7; no voters:
I20260925 11:12:10.892855 30512 leader_election.cc:288] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:10.893007 30522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:10.893868 30522 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [term 1 LEADER]: Becoming Leader. State: Replica: a78fa660119e4a638a790d31587350a7, State: Running, Role: LEADER
I20260925 11:12:10.894654 30522 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } }
I20260925 11:12:10.895501 30512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:10.898502 30523 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78fa660119e4a638a790d31587350a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } } }
I20260925 11:12:10.899207 30523 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:10.898530 30524 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78fa660119e4a638a790d31587350a7. Latest consensus state: current_term: 1 leader_uuid: "a78fa660119e4a638a790d31587350a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78fa660119e4a638a790d31587350a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41285 } } }
I20260925 11:12:10.901437 30524 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:10.903928 30528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:10.910760 30528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:10.922178 30528 catalog_manager.cc:1347] Generated new cluster ID: 24472b479ebc4f4d8aefae9a43799ddc
I20260925 11:12:10.922341 30528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:10.934186 30528 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:10.935858 30528 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:10.942596 30528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78fa660119e4a638a790d31587350a7: Generated new TSK 0
I20260925 11:12:10.943593 30528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:10.956215 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30445
W20260925 11:12:10.957543 30507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.176:54901
W20260925 11:12:10.958371 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30445: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:10.958565 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30445
W20260925 11:12:10.976044 30430 connection.cc:535] client connection to 127.0.0.1:41285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41285): Connection reset by peer (error 104)
2026-09-25T11:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:11 dist-test-slave-6042 krb5kdc[30543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:11 dist-test-slave-6042 krb5kdc[30543](info): set up 2 sockets
Sep 25 11:12:11 dist-test-slave-6042 krb5kdc[30543](info): commencing operation
krb5kdc: starting...
W20260925 11:12:12.079285 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:12 dist-test-slave-6042 krb5kdc[30543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 11:12:12.357561 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:12.706368 30556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:12.706815 30556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:12.706992 30556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:12.714537 30556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:12.714715 30556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:12.714823 30556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:12.714916 30556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:12.714985 30556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:12.725591 30556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:12.727037 30556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:12.728405 30556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:12.740538 30562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:12.740571 30561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:12.740585 30564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:12.741236 30556 server_base.cc:1034] running on GCE node
I20260925 11:12:12.742060 30556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:12.743508 30556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:12.744725 30556 hybrid_clock.cc:648] HybridClock initialized: now 1790334732744674 us; error 46 us; skew 500 ppm
I20260925 11:12:12.745178 30556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:12 dist-test-slave-6042 krb5kdc[30543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:12.748486 30556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 11:12:12.750083 30556 webserver.cc:460] Webserver started at http://127.0.0.1:41195/ using document root <none> and password file <none>
I20260925 11:12:12.750912 30556 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:12.751151 30556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:12.751703 30556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:12.755237 30556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "2e9696f985ba4912912ee49b44e32008"
format_stamp: "Formatted at 2026-09-25 11:12:12 on dist-test-slave-6042"
server_key: "d2bed4a176ec8277a9e4942b6f2a595d"
server_key_iv: "9058f4756a5aa154e53332bb01a8bddf"
server_key_version: "encryptionkey@0"
I20260925 11:12:12.756044 30556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "2e9696f985ba4912912ee49b44e32008"
format_stamp: "Formatted at 2026-09-25 11:12:12 on dist-test-slave-6042"
server_key: "d2bed4a176ec8277a9e4942b6f2a595d"
server_key_iv: "9058f4756a5aa154e53332bb01a8bddf"
server_key_version: "encryptionkey@0"
I20260925 11:12:12.761535 30556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 11:12:12.765271 30571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:12.766757 30556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:12:12.766947 30556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "2e9696f985ba4912912ee49b44e32008"
format_stamp: "Formatted at 2026-09-25 11:12:12 on dist-test-slave-6042"
server_key: "d2bed4a176ec8277a9e4942b6f2a595d"
server_key_iv: "9058f4756a5aa154e53332bb01a8bddf"
server_key_version: "encryptionkey@0"
I20260925 11:12:12.767217 30556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:12.773447 30556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:12.778851 30556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:12.779186 30556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:12.799819 30556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39033
I20260925 11:12:12.799875 30622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39033 every 8 connection(s)
I20260925 11:12:12.801286 30556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:12.805248 30623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:12.805353 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30556
I20260925 11:12:12.805526 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:12.806061 8916 external_mini_cluster.cc:1428] Setting key f894fe8b5cc6a85d83cebe0145007377
I20260925 11:12:12.814033 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:12.807895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:50801 (local address 127.0.0.1:39033)
0925 11:12:12.808753 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:12.808768 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:12.808811 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:12:12.810377 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:12.810400 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:12.810848 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:12.811233 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:12.812385 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:12.812393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:12.812425 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:12.812447 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:12.812511 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:12.812534 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:12.812735 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:12.812849 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 11:12:12.813083 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":402,"threads_started":1}
I20260925 11:12:12.817864 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:12.819514 30623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008: Bootstrap starting.
I20260925 11:12:12.823917 30623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:12.825428 30623 log.cc:824] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:12.825977 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:12.823320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:34689 (local address 127.0.0.1:39033)
0925 11:12:12.823528 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:12.823535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:12.823841 (+ 306us) server_negotiation.cc:407] Connection header received
0925 11:12:12.824055 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:12.824061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:12.824125 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:12.824215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:12.825252 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:12.825256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:12.825259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:12.825274 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:12.825314 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:12.825328 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:12.825424 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:12.825603 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 11:12:12.825671 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 11:12:12.828560 30623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008: No bootstrap required, opened a new log
I20260925 11:12:12.836642 30623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } }
I20260925 11:12:12.837059 30623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:12.837170 30623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9696f985ba4912912ee49b44e32008, State: Initialized, Role: FOLLOWER
I20260925 11:12:12.837792 30623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } }
I20260925 11:12:12.838019 30623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:12.838150 30623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:12.838327 30623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:12.840878 30623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } }
I20260925 11:12:12.841415 30623 leader_election.cc:302] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e9696f985ba4912912ee49b44e32008; no voters:
I20260925 11:12:12.842088 30623 leader_election.cc:288] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:12.842296 30633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:12.843082 30633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [term 1 LEADER]: Becoming Leader. State: Replica: 2e9696f985ba4912912ee49b44e32008, State: Running, Role: LEADER
I20260925 11:12:12.843659 30633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } }
I20260925 11:12:12.844302 30623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:12.846680 30635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e9696f985ba4912912ee49b44e32008. Latest consensus state: current_term: 1 leader_uuid: "2e9696f985ba4912912ee49b44e32008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } } }
I20260925 11:12:12.847384 30635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:12.846935 30634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e9696f985ba4912912ee49b44e32008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9696f985ba4912912ee49b44e32008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39033 } } }
I20260925 11:12:12.847934 30634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:12.850739 30642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:12.855947 30642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:12.865875 30642 catalog_manager.cc:1347] Generated new cluster ID: 9c37c321fa00433d9c0841fa07341c1c
I20260925 11:12:12.866019 30642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:12.891702 30642 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:12.892211 30642 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:12.903002 30642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e9696f985ba4912912ee49b44e32008: Generated new TSK 0
I20260925 11:12:12.903959 30642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:12.908325 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30556
W20260925 11:12:12.909551 30618 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.176:50801
W20260925 11:12:12.910328 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:12.910501 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30556
2026-09-25T11:12:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:13 dist-test-slave-6042 krb5kdc[30654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:13 dist-test-slave-6042 krb5kdc[30654](info): set up 2 sockets
Sep 25 11:12:13 dist-test-slave-6042 krb5kdc[30654](info): commencing operation
krb5kdc: starting...
W20260925 11:12:14.035532 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:14 dist-test-slave-6042 krb5kdc[30654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:14.318876 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34143
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:14.679057 30667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:14.679450 30667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:14.679620 30667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:14.687394 30667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:14.687573 30667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:14.687690 30667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:14.687798 30667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:14.687875 30667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:14.698688 30667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:14.700153 30667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:14.701612 30667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:14.714541 30672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:14.714501 30675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:14.714443 30673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:14.715754 30667 server_base.cc:1034] running on GCE node
I20260925 11:12:14.716670 30667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:14.718304 30667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:14.719527 30667 hybrid_clock.cc:648] HybridClock initialized: now 1790334734719475 us; error 67 us; skew 500 ppm
I20260925 11:12:14.719918 30667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:14 dist-test-slave-6042 krb5kdc[30654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334734, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:14.732761 30667 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:14.734743 30667 webserver.cc:460] Webserver started at http://127.8.181.62:38763/ using document root <none> and password file <none>
I20260925 11:12:14.735553 30667 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:14.735723 30667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:14.736066 30667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:14.739712 30667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "58d57aeec9e149b08345b2b097bd440f"
format_stamp: "Formatted at 2026-09-25 11:12:14 on dist-test-slave-6042"
server_key: "9b4ef4d20e58a85c6e1afd7eae40902d"
server_key_iv: "23835c81739812ff4801abcefebb2b68"
server_key_version: "encryptionkey@0"
I20260925 11:12:14.740615 30667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "58d57aeec9e149b08345b2b097bd440f"
format_stamp: "Formatted at 2026-09-25 11:12:14 on dist-test-slave-6042"
server_key: "9b4ef4d20e58a85c6e1afd7eae40902d"
server_key_iv: "23835c81739812ff4801abcefebb2b68"
server_key_version: "encryptionkey@0"
I20260925 11:12:14.746788 30667 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 11:12:14.750891 30682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:14.752575 30667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:12:14.752822 30667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "58d57aeec9e149b08345b2b097bd440f"
format_stamp: "Formatted at 2026-09-25 11:12:14 on dist-test-slave-6042"
server_key: "9b4ef4d20e58a85c6e1afd7eae40902d"
server_key_iv: "23835c81739812ff4801abcefebb2b68"
server_key_version: "encryptionkey@0"
I20260925 11:12:14.753059 30667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:14.775341 30667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:14.782425 30667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:14.782872 30667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:14.807650 30667 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34143
I20260925 11:12:14.807755 30733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34143 every 8 connection(s)
I20260925 11:12:14.809250 30667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:14.815333 30734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:14.817662 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30667
I20260925 11:12:14.817885 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:14.818526 8916 external_mini_cluster.cc:1428] Setting key b164def8247282764430d754846aba07
I20260925 11:12:14.831241 30734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f: Bootstrap starting.
Sep 25 11:12:14 dist-test-slave-6042 krb5kdc[30654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:14.835286 30734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:14.836565 30734 log.cc:824] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:14.840126 30734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f: No bootstrap required, opened a new log
I20260925 11:12:14.844936 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:14.820614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39111 (local address 127.8.181.62:34143)
0925 11:12:14.821683 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:14.821696 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:14.821749 (+ 53us) server_negotiation.cc:407] Connection header received
0925 11:12:14.824021 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:14.824063 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:14.824606 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:14.825072 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:14.826745 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:14.828227 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:14.829702 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:14.830526 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:14.830755 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:14.836194 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:14.836203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:14.836221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:14.836276 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:14.839963 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:14.840994 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:14.841002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:14.841008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:14.841188 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:14.841728 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:14.841732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:14.841734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:14.842409 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:14.842843 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:14.843396 (+ 553us) server_negotiation.cc:299] Negotiation successful
0925 11:12:14.843779 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":444,"threads_started":1}
I20260925 11:12:14.848837 30734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } }
I20260925 11:12:14.849308 30734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:14.849454 30734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d57aeec9e149b08345b2b097bd440f, State: Initialized, Role: FOLLOWER
I20260925 11:12:14.849500 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:14.850068 30734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } }
I20260925 11:12:14.850349 30734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:14.850546 30734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:14.850739 30734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:14.853232 30734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } }
I20260925 11:12:14.853941 30734 leader_election.cc:302] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58d57aeec9e149b08345b2b097bd440f; no voters:
I20260925 11:12:14.854652 30734 leader_election.cc:288] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:14.854813 30743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:14.857578 30734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:14.860065 30743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [term 1 LEADER]: Becoming Leader. State: Replica: 58d57aeec9e149b08345b2b097bd440f, State: Running, Role: LEADER
I20260925 11:12:14.860774 30743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } }
I20260925 11:12:14.866707 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58d57aeec9e149b08345b2b097bd440f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } } }
I20260925 11:12:14.867168 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:14.866271 30746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58d57aeec9e149b08345b2b097bd440f. Latest consensus state: current_term: 1 leader_uuid: "58d57aeec9e149b08345b2b097bd440f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d57aeec9e149b08345b2b097bd440f" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34143 } } }
I20260925 11:12:14.867974 30746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:14.887548 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:14.855485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33869 (local address 127.8.181.62:34143)
0925 11:12:14.855677 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:14.855685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:14.859319 (+ 3634us) server_negotiation.cc:407] Connection header received
0925 11:12:14.859588 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:14.859600 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:14.859711 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:14.859848 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:14.862509 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:14.864404 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:14.866945 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:14.867968 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:14.868109 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:14.872726 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:14.872738 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:14.872746 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:14.872810 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:14.882054 (+ 9244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:14.883749 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:14.883759 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:14.883765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:14.883982 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:14.884745 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:14.884754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:14.884760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:14.885370 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:14.885752 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:14.886688 (+ 936us) server_negotiation.cc:299] Negotiation successful
0925 11:12:14.886983 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:12:14.890240 30751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:14.897080 30751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:14.907829 30751 catalog_manager.cc:1347] Generated new cluster ID: 99eb0d0bbc944f14a0ea3d9e8c0041d7
I20260925 11:12:14.907989 30751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:14.926443 30751 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:14.928378 30751 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:14.934963 30751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58d57aeec9e149b08345b2b097bd440f: Generated new TSK 0
I20260925 11:12:14.936105 30751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:14.951299 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30667
I20260925 11:12:14.952646 30729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:15.051409 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30667
W20260925 11:12:15.069064 30652 connection.cc:535] client connection to 127.8.181.62:34143 recv error: Network error: failed to read from TLS socket (remote: 127.8.181.62:34143): Connection reset by peer (error 104)
2026-09-25T11:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:15 dist-test-slave-6042 krb5kdc[30767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:15 dist-test-slave-6042 krb5kdc[30767](info): set up 2 sockets
Sep 25 11:12:15 dist-test-slave-6042 krb5kdc[30767](info): commencing operation
krb5kdc: starting...
W20260925 11:12:16.179374 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:16 dist-test-slave-6042 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:16.467298 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36169
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:36169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:16.817019 30780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:16.817353 30780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:16.817525 30780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:16.825064 30780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:16.825207 30780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:16.825299 30780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:16.825416 30780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:16.825484 30780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:16.835865 30780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:36169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:36169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:16.837282 30780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:16.838680 30780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:16.851161 30785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:16.851413 30786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:16.851667 30788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:16.852656 30780 server_base.cc:1034] running on GCE node
I20260925 11:12:16.853550 30780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:16.855217 30780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:16.856480 30780 hybrid_clock.cc:648] HybridClock initialized: now 1790334736856396 us; error 81 us; skew 500 ppm
I20260925 11:12:16.856880 30780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:16 dist-test-slave-6042 krb5kdc[30767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334736, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:16.870886 30780 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:16.872897 30780 webserver.cc:460] Webserver started at http://127.8.181.62:35219/ using document root <none> and password file <none>
I20260925 11:12:16.873684 30780 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:16.873844 30780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:16.874173 30780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:16.877540 30780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "06a4b7380a6d41ed82794f9111099080"
format_stamp: "Formatted at 2026-09-25 11:12:16 on dist-test-slave-6042"
server_key: "fc9970d048aa711d1dcde61099eb86aa"
server_key_iv: "4e6fcf104c46c0dc4bbe0b10dc5a25dc"
server_key_version: "encryptionkey@0"
I20260925 11:12:16.878436 30780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "06a4b7380a6d41ed82794f9111099080"
format_stamp: "Formatted at 2026-09-25 11:12:16 on dist-test-slave-6042"
server_key: "fc9970d048aa711d1dcde61099eb86aa"
server_key_iv: "4e6fcf104c46c0dc4bbe0b10dc5a25dc"
server_key_version: "encryptionkey@0"
I20260925 11:12:16.884611 30780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 11:12:16.889161 30795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:16.890738 30780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 11:12:16.890931 30780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "06a4b7380a6d41ed82794f9111099080"
format_stamp: "Formatted at 2026-09-25 11:12:16 on dist-test-slave-6042"
server_key: "fc9970d048aa711d1dcde61099eb86aa"
server_key_iv: "4e6fcf104c46c0dc4bbe0b10dc5a25dc"
server_key_version: "encryptionkey@0"
I20260925 11:12:16.891161 30780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:16.904435 30780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:16.909157 30780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:16.909487 30780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:16.930059 30780 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:36169
I20260925 11:12:16.930166 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:36169 every 8 connection(s)
I20260925 11:12:16.931530 30780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:16.931830 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30780
I20260925 11:12:16.932010 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:16.932627 8916 external_mini_cluster.cc:1428] Setting key d6b35afa62805b3737e7cc3ab3c1ac80
I20260925 11:12:16.935822 30847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:16.946324 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:16.934512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51425 (local address 127.8.181.62:36169)
0925 11:12:16.935339 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:16.935350 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:16.935388 (+ 38us) server_negotiation.cc:407] Connection header received
0925 11:12:16.937073 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:16.937094 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:16.937529 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:16.937862 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:16.938457 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:16.939894 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:16.941729 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:16.942411 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:16.942671 (+ 260us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:16.943916 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:16.943924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:16.943937 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:16.944482 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:16.944567 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:16.944592 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:16.944713 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:16.945048 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 11:12:16.945302 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":375,"threads_started":1}
I20260925 11:12:16.948311 30847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080: Bootstrap starting.
I20260925 11:12:16.950155 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:16.952041 30847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:16.953408 30847 log.cc:824] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:16.956957 30847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080: No bootstrap required, opened a new log
I20260925 11:12:16.962771 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:16.955856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38229 (local address 127.8.181.62:36169)
0925 11:12:16.956065 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:16.956072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:16.956555 (+ 483us) server_negotiation.cc:407] Connection header received
0925 11:12:16.956712 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:16.956717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:16.956792 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:16.956906 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:16.957438 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:16.958616 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:16.959881 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:16.960602 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:16.960745 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:16.961838 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:16.961844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:16.961849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:16.961877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:16.961923 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:16.961943 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:16.962149 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:16.962326 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 11:12:16.962380 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 11:12:16.965919 30847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } }
I20260925 11:12:16.966323 30847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:16.966439 30847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a4b7380a6d41ed82794f9111099080, State: Initialized, Role: FOLLOWER
I20260925 11:12:16.967053 30847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } }
I20260925 11:12:16.967262 30847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:16.967360 30847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:16.967485 30847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:16.969815 30847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } }
I20260925 11:12:16.970371 30847 leader_election.cc:302] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a4b7380a6d41ed82794f9111099080; no voters:
I20260925 11:12:16.971145 30847 leader_election.cc:288] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:16.971249 30857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:16.972050 30857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [term 1 LEADER]: Becoming Leader. State: Replica: 06a4b7380a6d41ed82794f9111099080, State: Running, Role: LEADER
I20260925 11:12:16.972575 30857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } }
I20260925 11:12:16.973271 30847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:16.975554 30859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a4b7380a6d41ed82794f9111099080. Latest consensus state: current_term: 1 leader_uuid: "06a4b7380a6d41ed82794f9111099080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } } }
I20260925 11:12:16.975529 30858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a4b7380a6d41ed82794f9111099080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a4b7380a6d41ed82794f9111099080" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 36169 } } }
I20260925 11:12:16.975989 30859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:16.975993 30858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:16.977576 30863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:16.986277 30863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:16.996809 30863 catalog_manager.cc:1347] Generated new cluster ID: 89370a5c69c9467694f7607f7750efdb
I20260925 11:12:16.996949 30863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:17.021024 30863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:17.022536 30863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:17.032362 30863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a4b7380a6d41ed82794f9111099080: Generated new TSK 0
I20260925 11:12:17.033314 30863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:17.041677 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30780
W20260925 11:12:17.043005 30842 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51425
W20260925 11:12:17.043804 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30780: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:17.043956 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30780
W20260925 11:12:17.058163 30765 connection.cc:535] client connection to 127.8.181.62:36169 recv error: Network error: failed to read from TLS socket (remote: 127.8.181.62:36169): Connection reset by peer (error 104)
2026-09-25T11:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:17 dist-test-slave-6042 krb5kdc[30878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:17 dist-test-slave-6042 krb5kdc[30878](info): set up 2 sockets
Sep 25 11:12:17 dist-test-slave-6042 krb5kdc[30878](info): commencing operation
krb5kdc: starting...
W20260925 11:12:18.187273 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:18 dist-test-slave-6042 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:18.471376 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40983
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:40983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:18.842218 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:18.842645 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:18.842847 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:18.850654 30891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:18.850793 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:18.850922 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:18.851047 30891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:18.851130 30891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:18.861959 30891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:40983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:40983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:18.863444 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:18.864841 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:18.878268 30897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:18.878356 30899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:18.878350 30896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:18.879781 30891 server_base.cc:1034] running on GCE node
I20260925 11:12:18.880789 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:18.882787 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:18.884128 30891 hybrid_clock.cc:648] HybridClock initialized: now 1790334738884048 us; error 53 us; skew 500 ppm
I20260925 11:12:18.884557 30891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:18 dist-test-slave-6042 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334738, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:18.888265 30891 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:18.890002 30891 webserver.cc:460] Webserver started at http://127.8.181.62:33351/ using document root <none> and password file <none>
I20260925 11:12:18.890913 30891 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:18.891105 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:18.891574 30891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:18.896049 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "16e5a0ce51bc49008f3a47c844df6a08"
format_stamp: "Formatted at 2026-09-25 11:12:18 on dist-test-slave-6042"
server_key: "6597aaada984cf704ddae72cfb806dc4"
server_key_iv: "ae7ecc68e035269293c09fb15e2bb34e"
server_key_version: "encryptionkey@0"
I20260925 11:12:18.896914 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "16e5a0ce51bc49008f3a47c844df6a08"
format_stamp: "Formatted at 2026-09-25 11:12:18 on dist-test-slave-6042"
server_key: "6597aaada984cf704ddae72cfb806dc4"
server_key_iv: "ae7ecc68e035269293c09fb15e2bb34e"
server_key_version: "encryptionkey@0"
I20260925 11:12:18.903245 30891 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 11:12:18.907357 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:18.909056 30891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:12:18.909279 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "16e5a0ce51bc49008f3a47c844df6a08"
format_stamp: "Formatted at 2026-09-25 11:12:18 on dist-test-slave-6042"
server_key: "6597aaada984cf704ddae72cfb806dc4"
server_key_iv: "ae7ecc68e035269293c09fb15e2bb34e"
server_key_version: "encryptionkey@0"
I20260925 11:12:18.909497 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:18.916184 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:18.920877 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:18.921223 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:18.941529 30891 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:40983
I20260925 11:12:18.941665 30957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:40983 every 8 connection(s)
I20260925 11:12:18.943111 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:18.947643 30958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:18.951350 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 30891
I20260925 11:12:18.951547 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:18.952111 8916 external_mini_cluster.cc:1428] Setting key 4fbd808783aee55a67f0cd06d1aa47ee
I20260925 11:12:18.961154 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:18.954317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58785 (local address 127.8.181.62:40983)
0925 11:12:18.955313 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:18.955327 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:18.955372 (+ 45us) server_negotiation.cc:407] Connection header received
0925 11:12:18.957166 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:18.957193 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:18.957610 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:18.958149 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:18.959380 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:18.959387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:18.959402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:18.959416 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:18.959478 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:18.959498 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:18.959754 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:18.959907 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 11:12:18.960154 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":394,"threads_started":1}
I20260925 11:12:18.961804 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08: Bootstrap starting.
I20260925 11:12:18.965204 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:18.965618 30958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:18.966832 30958 log.cc:824] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:18.970513 30958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08: No bootstrap required, opened a new log
I20260925 11:12:18.973688 30961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:18.970422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39709 (local address 127.8.181.62:40983)
0925 11:12:18.970703 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:18.970710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:18.971294 (+ 584us) server_negotiation.cc:407] Connection header received
0925 11:12:18.971523 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:18.971531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:18.971594 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:18.971715 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:18.972816 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:18.972824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:18.972834 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:18.972853 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:18.972893 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:18.972910 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:18.973153 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:18.973281 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 11:12:18.973358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:12:18.979393 30958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } }
I20260925 11:12:18.979962 30958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:18.980103 30958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e5a0ce51bc49008f3a47c844df6a08, State: Initialized, Role: FOLLOWER
I20260925 11:12:18.980792 30958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } }
I20260925 11:12:18.981105 30958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:18.981271 30958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:18.981460 30958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:18.983960 30958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } }
I20260925 11:12:18.984633 30958 leader_election.cc:302] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e5a0ce51bc49008f3a47c844df6a08; no voters:
I20260925 11:12:18.985433 30958 leader_election.cc:288] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:18.985554 30968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:18.986231 30968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [term 1 LEADER]: Becoming Leader. State: Replica: 16e5a0ce51bc49008f3a47c844df6a08, State: Running, Role: LEADER
I20260925 11:12:18.986908 30968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } }
I20260925 11:12:18.987998 30958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:18.989979 30969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e5a0ce51bc49008f3a47c844df6a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } } }
I20260925 11:12:18.990533 30969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:18.991595 30970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e5a0ce51bc49008f3a47c844df6a08. Latest consensus state: current_term: 1 leader_uuid: "16e5a0ce51bc49008f3a47c844df6a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e5a0ce51bc49008f3a47c844df6a08" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 40983 } } }
I20260925 11:12:18.992173 30970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:18.992600 30974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:18.998829 30974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:19.009635 30974 catalog_manager.cc:1347] Generated new cluster ID: 2cb9140f8ec644df8198f83b76685bb2
I20260925 11:12:19.009775 30974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:19.038924 30974 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:19.039640 30974 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:19.048230 30974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e5a0ce51bc49008f3a47c844df6a08: Generated new TSK 0
I20260925 11:12:19.049299 30974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:19.054992 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30891
W20260925 11:12:19.056602 30953 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58785
W20260925 11:12:19.057441 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 30891: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:19.057662 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 30891
2026-09-25T11:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:19 dist-test-slave-6042 krb5kdc[30989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:19 dist-test-slave-6042 krb5kdc[30989](info): set up 2 sockets
Sep 25 11:12:19 dist-test-slave-6042 krb5kdc[30989](info): commencing operation
krb5kdc: starting...
W20260925 11:12:20.219383 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:20 dist-test-slave-6042 krb5kdc[30989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:20.507237 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43347
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:20.858338 31002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:20.858754 31002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:20.858856 31002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:20.866550 31002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:20.866698 31002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:20.866747 31002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:20.866812 31002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:20.866858 31002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:20.877314 31002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:20.878790 31002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:20.880199 31002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:20.892895 31007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:20.893134 31008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:20.893254 31010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:20.894548 31002 server_base.cc:1034] running on GCE node
I20260925 11:12:20.895414 31002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:20.897150 31002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:20.898379 31002 hybrid_clock.cc:648] HybridClock initialized: now 1790334740898339 us; error 34 us; skew 500 ppm
I20260925 11:12:20.898840 31002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:20 dist-test-slave-6042 krb5kdc[30989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334740, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:20.903781 31002 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:20.905578 31002 webserver.cc:460] Webserver started at http://127.8.181.62:38129/ using document root <none> and password file <none>
I20260925 11:12:20.906363 31002 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:20.906546 31002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:20.906915 31002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:20.910100 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "4b822429be144c119d0c840f855736ec"
format_stamp: "Formatted at 2026-09-25 11:12:20 on dist-test-slave-6042"
server_key: "6263d574f0d97f82fad54fbf51ee8a13"
server_key_iv: "0af7c5bac806fd3dd7c40c6f55e9a9c7"
server_key_version: "encryptionkey@0"
I20260925 11:12:20.911053 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "4b822429be144c119d0c840f855736ec"
format_stamp: "Formatted at 2026-09-25 11:12:20 on dist-test-slave-6042"
server_key: "6263d574f0d97f82fad54fbf51ee8a13"
server_key_iv: "0af7c5bac806fd3dd7c40c6f55e9a9c7"
server_key_version: "encryptionkey@0"
I20260925 11:12:20.916962 31002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 11:12:20.920969 31017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:20.922518 31002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 11:12:20.922755 31002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "4b822429be144c119d0c840f855736ec"
format_stamp: "Formatted at 2026-09-25 11:12:20 on dist-test-slave-6042"
server_key: "6263d574f0d97f82fad54fbf51ee8a13"
server_key_iv: "0af7c5bac806fd3dd7c40c6f55e9a9c7"
server_key_version: "encryptionkey@0"
I20260925 11:12:20.922957 31002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:20.945255 31002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:20.950215 31002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:20.950611 31002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:20.970453 31002 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:43347
I20260925 11:12:20.970501 31068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:43347 every 8 connection(s)
I20260925 11:12:20.972057 31002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:20.976171 31069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:20.977481 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31002
I20260925 11:12:20.977675 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:20.978231 8916 external_mini_cluster.cc:1428] Setting key 4849ff5edaf355a8d0ff65957bc4a039
I20260925 11:12:20.991518 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec: Bootstrap starting.
Sep 25 11:12:20 dist-test-slave-6042 krb5kdc[30989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:20.995450 31069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:20.996785 31069 log.cc:824] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:21.000032 31069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec: No bootstrap required, opened a new log
I20260925 11:12:21.003293 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:20.980313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41637 (local address 127.8.181.62:43347)
0925 11:12:20.981200 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:20.981212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:20.981258 (+ 46us) server_negotiation.cc:407] Connection header received
0925 11:12:20.983059 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:20.983090 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:20.983550 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:20.983919 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:20.985813 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:20.987239 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:20.989121 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:20.989834 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:20.989979 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:20.995085 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:20.995094 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:20.995112 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:20.995154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:20.998633 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:20.999480 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:20.999500 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:20.999508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:20.999684 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:21.000102 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:21.000108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:21.000111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:21.000737 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:21.001085 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:21.001569 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 11:12:21.002081 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":430,"threads_started":1}
I20260925 11:12:21.008188 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:21.009790 31069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } }
I20260925 11:12:21.010212 31069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:21.010344 31069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b822429be144c119d0c840f855736ec, State: Initialized, Role: FOLLOWER
I20260925 11:12:21.011166 31069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } }
I20260925 11:12:21.011482 31069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:21.011636 31069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:21.011840 31069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:21.014850 31069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } }
I20260925 11:12:21.015439 31069 leader_election.cc:302] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b822429be144c119d0c840f855736ec; no voters:
I20260925 11:12:21.016230 31069 leader_election.cc:288] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:21.016361 31079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:21.018700 31069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:21.018831 31079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [term 1 LEADER]: Becoming Leader. State: Replica: 4b822429be144c119d0c840f855736ec, State: Running, Role: LEADER
I20260925 11:12:21.019477 31079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } }
I20260925 11:12:21.033353 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:21.012843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37763 (local address 127.8.181.62:43347)
0925 11:12:21.013055 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:21.013062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:21.013572 (+ 510us) server_negotiation.cc:407] Connection header received
0925 11:12:21.013728 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:21.013734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:21.013819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:21.013945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:21.015538 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:21.016757 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:21.018676 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:21.019455 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:21.019606 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:21.023710 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:21.023718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:21.023723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:21.023768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:21.029074 (+ 5306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:21.030164 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:21.030173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:21.030180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:21.030384 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:21.031056 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:21.031064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:21.031070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:21.031665 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:21.032025 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:21.032457 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 11:12:21.032785 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 11:12:21.035017 31081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b822429be144c119d0c840f855736ec. Latest consensus state: current_term: 1 leader_uuid: "4b822429be144c119d0c840f855736ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } } }
I20260925 11:12:21.035372 31081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:21.035969 31080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b822429be144c119d0c840f855736ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b822429be144c119d0c840f855736ec" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 43347 } } }
I20260925 11:12:21.036288 31080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:21.036319 31095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:21.041922 31095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:21.052767 31095 catalog_manager.cc:1347] Generated new cluster ID: e340b9bcd3644f8a84608fa37fab651a
I20260925 11:12:21.052906 31095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:21.079459 31095 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:21.082809 31095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:21.092450 31095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b822429be144c119d0c840f855736ec: Generated new TSK 0
I20260925 11:12:21.093739 31095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:21.107398 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31002
I20260925 11:12:21.108601 31064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:21.207450 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31002
2026-09-25T11:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:21 dist-test-slave-6042 krb5kdc[31102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:21 dist-test-slave-6042 krb5kdc[31102](info): set up 2 sockets
Sep 25 11:12:21 dist-test-slave-6042 krb5kdc[31102](info): commencing operation
krb5kdc: starting...
W20260925 11:12:22.343370 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:22 dist-test-slave-6042 krb5kdc[31102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:22.644248 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34631
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:34631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:23.007791 31115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:23.008212 31115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:23.008391 31115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:23.016149 31115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:23.016363 31115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:23.016485 31115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:23.016601 31115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:23.016680 31115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:23.027580 31115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:34631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:34631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:23.029106 31115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:23.030589 31115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:23.044194 31120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:23.044198 31121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:23.044243 31123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:23.045303 31115 server_base.cc:1034] running on GCE node
I20260925 11:12:23.046235 31115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:23.047986 31115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:23.049254 31115 hybrid_clock.cc:648] HybridClock initialized: now 1790334743049214 us; error 56 us; skew 500 ppm
I20260925 11:12:23.049726 31115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:23 dist-test-slave-6042 krb5kdc[31102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334743, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:23.059396 31115 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:23.061283 31115 webserver.cc:460] Webserver started at http://127.8.181.62:36533/ using document root <none> and password file <none>
I20260925 11:12:23.062162 31115 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:23.062325 31115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:23.062759 31115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:23.066406 31115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47"
format_stamp: "Formatted at 2026-09-25 11:12:23 on dist-test-slave-6042"
server_key: "c554ac47ae1be89b18b07f44217db3d9"
server_key_iv: "3040a5d9c409fe62a4f7208eb391042f"
server_key_version: "encryptionkey@0"
I20260925 11:12:23.067442 31115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47"
format_stamp: "Formatted at 2026-09-25 11:12:23 on dist-test-slave-6042"
server_key: "c554ac47ae1be89b18b07f44217db3d9"
server_key_iv: "3040a5d9c409fe62a4f7208eb391042f"
server_key_version: "encryptionkey@0"
I20260925 11:12:23.073616 31115 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 11:12:23.077991 31130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:23.079783 31115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:12:23.080026 31115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47"
format_stamp: "Formatted at 2026-09-25 11:12:23 on dist-test-slave-6042"
server_key: "c554ac47ae1be89b18b07f44217db3d9"
server_key_iv: "3040a5d9c409fe62a4f7208eb391042f"
server_key_version: "encryptionkey@0"
I20260925 11:12:23.080279 31115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:23.101300 31115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:23.106374 31115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:23.106762 31115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:23.128252 31115 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:34631
I20260925 11:12:23.128278 31181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:34631 every 8 connection(s)
I20260925 11:12:23.129830 31115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:23.133845 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:23.134784 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31115
I20260925 11:12:23.134985 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:23.135537 8916 external_mini_cluster.cc:1428] Setting key ef7e866d8431c2b1329a556e0b5799f3
I20260925 11:12:23.149435 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: Bootstrap starting.
I20260925 11:12:23.151324 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:23.137646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51061 (local address 127.8.181.62:34631)
0925 11:12:23.138620 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:23.138633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:23.138676 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:12:23.140637 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:23.140666 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:23.141143 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:23.141529 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:23.142304 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:23.143742 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:23.145850 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:23.146813 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:23.146972 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:23.148469 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:23.148478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:23.148496 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:23.149190 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:23.149285 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:23.149316 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:23.149414 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:23.149861 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 11:12:23.150164 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":435,"threads_started":1}
I20260925 11:12:23.153371 31182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:23.155212 31182 log.cc:824] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:23.155750 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:23.158221 31182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: No bootstrap required, opened a new log
I20260925 11:12:23.168314 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:23.160492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46431 (local address 127.8.181.62:34631)
0925 11:12:23.160699 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:23.160707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:23.161310 (+ 603us) server_negotiation.cc:407] Connection header received
0925 11:12:23.161502 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:23.161507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:23.161579 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:23.161686 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:23.162367 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:23.163291 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:23.165332 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:23.165885 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:23.165999 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:23.167311 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:23.167317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:23.167320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:23.167345 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:23.167391 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:23.167408 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:23.167508 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:23.167791 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 11:12:23.167877 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 11:12:23.168970 31182 raft_consensus.cc:348] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } }
I20260925 11:12:23.169524 31182 raft_consensus.cc:374] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:23.169641 31182 raft_consensus.cc:729] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d842a9f769dd4fa4a6fb5ebc9ba68a47, State: Initialized, Role: FOLLOWER
I20260925 11:12:23.170301 31182 consensus_queue.cc:260] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } }
I20260925 11:12:23.170642 31182 raft_consensus.cc:388] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:23.170799 31182 raft_consensus.cc:482] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:23.171008 31182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:23.173311 31182 raft_consensus.cc:504] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } }
I20260925 11:12:23.173877 31182 leader_election.cc:302] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d842a9f769dd4fa4a6fb5ebc9ba68a47; no voters:
I20260925 11:12:23.174623 31182 leader_election.cc:288] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:23.174795 31192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:23.176470 31192 raft_consensus.cc:686] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [term 1 LEADER]: Becoming Leader. State: Replica: d842a9f769dd4fa4a6fb5ebc9ba68a47, State: Running, Role: LEADER
I20260925 11:12:23.177075 31192 consensus_queue.cc:237] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } }
I20260925 11:12:23.177238 31182 sys_catalog.cc:564] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:23.180962 31193 sys_catalog.cc:455] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } } }
I20260925 11:12:23.181468 31193 sys_catalog.cc:458] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:23.181886 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d842a9f769dd4fa4a6fb5ebc9ba68a47. Latest consensus state: current_term: 1 leader_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d842a9f769dd4fa4a6fb5ebc9ba68a47" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 34631 } } }
I20260925 11:12:23.182185 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47 [sys.catalog]: This master's current role is: LEADER
W20260925 11:12:23.190328 31208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:12:23.190531 31208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:12:23.190678 31201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:23.192986 31201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:23.203238 31201 catalog_manager.cc:1347] Generated new cluster ID: 0017b62e86e34904a705a055fb26ec13
I20260925 11:12:23.203373 31201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:23.224202 31201 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:23.225772 31201 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:23.243139 31201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d842a9f769dd4fa4a6fb5ebc9ba68a47: Generated new TSK 0
I20260925 11:12:23.244220 31201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:23.251672 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31115
W20260925 11:12:23.253026 31177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51061
W20260925 11:12:23.253851 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:23.254067 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31115
2026-09-25T11:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:23 dist-test-slave-6042 krb5kdc[31213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:23 dist-test-slave-6042 krb5kdc[31213](info): set up 2 sockets
Sep 25 11:12:23 dist-test-slave-6042 krb5kdc[31213](info): commencing operation
krb5kdc: starting...
W20260925 11:12:24.395541 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:24 dist-test-slave-6042 krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:24.696146 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41105
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:33529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:25.067561 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:25.067889 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:25.068121 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:25.075747 31226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:25.075881 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:25.075994 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:25.076135 31226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:25.076225 31226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:25.086948 31226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:33529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:41105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:25.088471 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:25.090128 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:25.104180 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:25.104229 31232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:25.105003 31234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:25.105435 31226 server_base.cc:1034] running on GCE node
I20260925 11:12:25.106448 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:25.108182 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:25.109514 31226 hybrid_clock.cc:648] HybridClock initialized: now 1790334745109442 us; error 58 us; skew 500 ppm
I20260925 11:12:25.109987 31226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:25 dist-test-slave-6042 krb5kdc[31213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334745, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:25.113724 31226 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:25.115639 31226 webserver.cc:460] Webserver started at http://127.8.181.62:39085/ using document root <none> and password file <none>
I20260925 11:12:25.116474 31226 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:25.116647 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:25.117004 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:25.120452 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "247a79c9613d40ce91db64ccde855875"
format_stamp: "Formatted at 2026-09-25 11:12:25 on dist-test-slave-6042"
server_key: "2a36ad27c1244c5e118d501e366ccb23"
server_key_iv: "bc64b7d1fdc439e0cd2c9bcffaa38219"
server_key_version: "encryptionkey@0"
I20260925 11:12:25.121313 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "247a79c9613d40ce91db64ccde855875"
format_stamp: "Formatted at 2026-09-25 11:12:25 on dist-test-slave-6042"
server_key: "2a36ad27c1244c5e118d501e366ccb23"
server_key_iv: "bc64b7d1fdc439e0cd2c9bcffaa38219"
server_key_version: "encryptionkey@0"
I20260925 11:12:25.127251 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 11:12:25.131691 31241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:25.133366 31226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260925 11:12:25.133606 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "247a79c9613d40ce91db64ccde855875"
format_stamp: "Formatted at 2026-09-25 11:12:25 on dist-test-slave-6042"
server_key: "2a36ad27c1244c5e118d501e366ccb23"
server_key_iv: "bc64b7d1fdc439e0cd2c9bcffaa38219"
server_key_version: "encryptionkey@0"
I20260925 11:12:25.133854 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:25.140938 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:25.145829 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:25.146183 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:25.166926 31226 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:41105
I20260925 11:12:25.166927 31292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:41105 every 8 connection(s)
I20260925 11:12:25.168789 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:25.173050 31293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:25.177137 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31226
I20260925 11:12:25.177335 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:25.177897 8916 external_mini_cluster.cc:1428] Setting key 001c870deb0e66743ba77a341c46e109
I20260925 11:12:25.187182 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:25.180087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60487 (local address 127.8.181.62:41105)
0925 11:12:25.181009 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:25.181023 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:25.181070 (+ 47us) server_negotiation.cc:407] Connection header received
0925 11:12:25.183023 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:25.183056 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:25.183491 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:25.183984 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:25.185294 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:25.185300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:25.185315 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:25.185332 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:25.185395 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:25.185415 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:25.185683 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:25.185839 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 11:12:25.186055 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":427,"threads_started":1}
I20260925 11:12:25.187314 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875: Bootstrap starting.
I20260925 11:12:25.190997 31293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:25.191448 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:25.192222 31293 log.cc:824] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:25.195760 31293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875: No bootstrap required, opened a new log
I20260925 11:12:25.198994 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:25.196105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48235 (local address 127.8.181.62:41105)
0925 11:12:25.196335 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:25.196348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:25.196842 (+ 494us) server_negotiation.cc:407] Connection header received
0925 11:12:25.197001 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:25.197006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:25.197074 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:25.197219 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:25.198204 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:25.198211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:25.198216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 11:12:25.198244 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:25.198302 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 11:12:25.198322 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:25.198457 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:25.198608 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 11:12:25.198673 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:12:25.204679 31293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } }
I20260925 11:12:25.205089 31293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 11:12:25.205237 31293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247a79c9613d40ce91db64ccde855875, State: Initialized, Role: FOLLOWER
I20260925 11:12:25.205945 31293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } }
I20260925 11:12:25.206223 31293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 11:12:25.206386 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 11:12:25.206615 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:25.208930 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } }
I20260925 11:12:25.209501 31293 leader_election.cc:302] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247a79c9613d40ce91db64ccde855875; no voters:
I20260925 11:12:25.210234 31293 leader_election.cc:288] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 11:12:25.210404 31303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:25.211285 31303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [term 1 LEADER]: Becoming Leader. State: Replica: 247a79c9613d40ce91db64ccde855875, State: Running, Role: LEADER
I20260925 11:12:25.211898 31303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } }
I20260925 11:12:25.212565 31293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:25.215716 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "247a79c9613d40ce91db64ccde855875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } } }
I20260925 11:12:25.215723 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247a79c9613d40ce91db64ccde855875. Latest consensus state: current_term: 1 leader_uuid: "247a79c9613d40ce91db64ccde855875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247a79c9613d40ce91db64ccde855875" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 41105 } } }
I20260925 11:12:25.216245 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:25.216398 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:25.219249 31310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:25.225333 31310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:25.237423 31310 catalog_manager.cc:1347] Generated new cluster ID: ccfd0ec060b04a5aabcedf0f2c864b01
I20260925 11:12:25.237594 31310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:25.249723 31310 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:25.250375 31310 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:25.271311 31310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 247a79c9613d40ce91db64ccde855875: Generated new TSK 0
I20260925 11:12:25.272616 31310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:25.291957 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31226
W20260925 11:12:25.293473 31288 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60487
W20260925 11:12:25.294498 8916 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31226: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 11:12:25.294660 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31226
2026-09-25T11:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2040 ms)
[----------] 18 tests from AuthTokenIssuingTest (36868 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:25 dist-test-slave-6042 krb5kdc[31324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:25 dist-test-slave-6042 krb5kdc[31324](info): set up 2 sockets
Sep 25 11:12:25 dist-test-slave-6042 krb5kdc[31324](info): commencing operation
krb5kdc: starting...
W20260925 11:12:26.427520 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:26 dist-test-slave-6042 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:26.724268 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44613
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:27.086788 31337 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:27.087205 31337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:27.087332 31337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:27.087479 31337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:27.095602 31337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:27.095714 31337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:27.095803 31337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:27.095911 31337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:27.096014 31337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:27.106735 31337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:27.108175 31337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:27.109557 31337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:27.122854 31342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:27.122824 31343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:27.122910 31345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:27.123603 31337 server_base.cc:1034] running on GCE node
I20260925 11:12:27.124486 31337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:27.126116 31337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:27.127410 31337 hybrid_clock.cc:648] HybridClock initialized: now 1790334747127340 us; error 50 us; skew 500 ppm
I20260925 11:12:27.127808 31337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:27.131351 31337 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:27.133023 31337 webserver.cc:460] Webserver started at http://127.8.181.62:34987/ using document root <none> and password file <none>
I20260925 11:12:27.133723 31337 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:27.133814 31337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:27.134083 31337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:27.137326 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "5a81cd8e793747398d25420af082e983"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "901c51a4991df877ae9736809672f952"
server_key_iv: "753ee4356b97eff45fb8de6f80a4c954"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.138163 31337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "5a81cd8e793747398d25420af082e983"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "901c51a4991df877ae9736809672f952"
server_key_iv: "753ee4356b97eff45fb8de6f80a4c954"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.143958 31337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 11:12:27.147691 31352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:27.149222 31337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:12:27.149437 31337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "5a81cd8e793747398d25420af082e983"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "901c51a4991df877ae9736809672f952"
server_key_iv: "753ee4356b97eff45fb8de6f80a4c954"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.149709 31337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:27.162396 31337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:27.167261 31337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:27.167574 31337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:27.187814 31337 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:44613
I20260925 11:12:27.187860 31403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:44613 every 8 connection(s)
I20260925 11:12:27.189308 31337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:27.193428 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31337
I20260925 11:12:27.193519 31404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:27.193672 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:27.194499 8916 external_mini_cluster.cc:1428] Setting key ba367b8eb337d25d84bd1caabc58d378
I20260925 11:12:27.204347 31404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.61@KRBTEST.COM" created.
I20260925 11:12:27.233057 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.208439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.8.181.62:44613)
0925 11:12:27.209425 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:27.209442 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:27.210226 (+ 784us) server_negotiation.cc:407] Connection header received
0925 11:12:27.212630 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:27.212641 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:27.212994 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:27.213217 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:27.214756 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.216788 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.218426 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.219526 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.219742 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.223905 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:27.223917 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:27.223935 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:27.223977 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:27.227834 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.228985 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.228993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.228995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.229177 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.229674 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.229677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.229679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.230240 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:27.230404 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:27.231214 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 11:12:27.231568 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":438,"threads_started":1}
I20260925 11:12:27.233093 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.208439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44613 (local address 127.0.0.1:35264)
0925 11:12:27.209996 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.210043 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:27.212156 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:27.213306 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:27.213312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:27.213963 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:27.214429 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.214450 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.216984 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.216988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.218251 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.218265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.219846 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.219857 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.220032 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.220867 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:27.220893 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:27.223601 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:27.228023 (+ 4422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.228131 (+ 108us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.228149 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.228791 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.229310 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.229318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.229325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.229554 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.230408 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:27.230420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:27.230713 (+ 293us) client_negotiation.cc:769] Sending connection context
0925 11:12:27.231039 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 11:12:27.231805 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":194,"thread_start_us":882,"threads_started":1}
I20260925 11:12:27.236937 31404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:27.238226 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.237703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37228)
0925 11:12:27.237932 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.238092 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 11:12:27.238549 31355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
I20260925 11:12:27.240345 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.239834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37236)
0925 11:12:27.240039 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.240139 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 11:12:27.240566 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38693: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
I20260925 11:12:27.262709 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:38693
--webserver_interface=127.8.181.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:12:27.289973 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } attempt: 1
I20260925 11:12:27.291332 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.290773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37242)
0925 11:12:27.291018 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.291165 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 11:12:27.292527 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.292072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37256)
0925 11:12:27.292236 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.292326 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 11:12:27.292924 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38693: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
I20260925 11:12:27.361235 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } attempt: 2
I20260925 11:12:27.362609 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.362000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37262)
0925 11:12:27.362256 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.362374 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 11:12:27.363917 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.363395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37268)
0925 11:12:27.363609 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.363732 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 11:12:27.364216 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38693: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
I20260925 11:12:27.455529 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } attempt: 3
I20260925 11:12:27.456900 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.456278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37274)
0925 11:12:27.456574 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.456705 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 11:12:27.458127 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.457669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37282)
0925 11:12:27.457844 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.457938 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 11:12:27.458400 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38693: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
I20260925 11:12:27.601754 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } attempt: 4
I20260925 11:12:27.603262 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.602635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37290)
0925 11:12:27.602976 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.603090 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 11:12:27.604454 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.604005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37306)
0925 11:12:27.604192 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.604297 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 11:12:27.604712 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38693: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38693: connect: Connection refused (error 111)
W20260925 11:12:27.621060 31411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:27.621450 31411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:27.621554 31411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:27.621726 31411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:27.629618 31411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:27.629725 31411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:27.629814 31411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:27.629890 31411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:27.629983 31411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:27.640554 31411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.181.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:27.642051 31411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:27.643509 31411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:27.657425 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:27.657510 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:27.657401 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:27.658347 31411 server_base.cc:1034] running on GCE node
I20260925 11:12:27.659283 31411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:27.660912 31411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:27.662192 31411 hybrid_clock.cc:648] HybridClock initialized: now 1790334747662117 us; error 37 us; skew 500 ppm
I20260925 11:12:27.662670 31411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:27.666674 31411 init.cc:377] Logged in from keytab as kudu/127.8.181.61@KRBTEST.COM (short username kudu)
I20260925 11:12:27.668430 31411 webserver.cc:460] Webserver started at http://127.8.181.61:43163/ using document root <none> and password file <none>
I20260925 11:12:27.669209 31411 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:27.669312 31411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:27.669613 31411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:27.673444 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data/instance:
uuid: "8ee75bf2f52c45debb92c220c3d11726"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "083c8df55f77d996c925b083bd547d60"
server_key_iv: "eed82d33264c4f09de29811a496a1876"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.674391 31411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal/instance:
uuid: "8ee75bf2f52c45debb92c220c3d11726"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "083c8df55f77d996c925b083bd547d60"
server_key_iv: "eed82d33264c4f09de29811a496a1876"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.681602 31411 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260925 11:12:27.686072 31426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:27.688030 31411 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 11:12:27.688265 31411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "8ee75bf2f52c45debb92c220c3d11726"
format_stamp: "Formatted at 2026-09-25 11:12:27 on dist-test-slave-6042"
server_key: "083c8df55f77d996c925b083bd547d60"
server_key_iv: "eed82d33264c4f09de29811a496a1876"
server_key_version: "encryptionkey@0"
I20260925 11:12:27.688549 31411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:27.702731 31411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:27.707563 31411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:27.707911 31411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:27.728682 31411 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.61:38693
I20260925 11:12:27.728729 31477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.61:38693 every 8 connection(s)
I20260925 11:12:27.730177 31411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
I20260925 11:12:27.731804 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31411
I20260925 11:12:27.731985 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-1/wal/instance
I20260925 11:12:27.732425 8916 external_mini_cluster.cc:1428] Setting key 2216a7df755df3bce30f9aa9977e574a
I20260925 11:12:27.734728 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:27.742681 31478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.181.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.60@KRBTEST.COM" created.
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:27.773878 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.746590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.8.181.62:44613)
0925 11:12:27.747141 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:27.747148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:27.748410 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 11:12:27.751827 (+ 3417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:27.751832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:27.751917 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:27.752036 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:27.754618 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.755378 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.758283 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.758844 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.758973 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.766707 (+ 7734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:27.766713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:27.766717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:27.766778 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:27.769744 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.771081 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.771086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.771089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.771242 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.771815 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.771820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.771823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.772129 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:27.772254 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:27.773351 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0925 11:12:27.773547 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260925 11:12:27.775547 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.747285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44613 (local address 127.0.0.1:35274)
0925 11:12:27.748190 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.748244 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:27.751382 (+ 3138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:27.752376 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:27.752393 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:27.753331 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:27.754283 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.754316 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.755594 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.755602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.758011 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.758051 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.759150 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.759160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.759532 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.760446 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:27.760487 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:27.766305 (+ 5818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:27.769965 (+ 3660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.769992 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.770017 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.770886 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.771378 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.771386 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.771393 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.771674 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.772283 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:27.772294 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:27.772840 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 11:12:27.773204 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 11:12:27.773812 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":350,"threads_started":1}
I20260925 11:12:27.777792 31478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:27.797866 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.778807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37314)
0925 11:12:27.779018 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.779043 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:27.779190 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:27.780342 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:27.780345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:27.780821 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:27.780973 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.780991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.783289 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.783294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.784220 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.784235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.785673 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.785678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.785822 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.786589 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:27.786608 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:27.789097 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:27.794450 (+ 5353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.794455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.794458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.795022 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.795835 (+ 813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.795838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.795841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.795984 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.797086 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:27.797092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:27.797231 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 11:12:27.797349 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 11:12:27.797596 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 11:12:27.798661 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.778972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.8.181.61:38693)
0925 11:12:27.779748 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:27.779759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:27.779794 (+ 35us) server_negotiation.cc:407] Connection header received
0925 11:12:27.779857 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:27.779868 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:27.780141 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:27.780320 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:27.781195 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.783076 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.784401 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.785475 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.785639 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.789386 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:27.789410 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:27.789434 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:27.789485 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:27.794198 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.795514 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.795522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.795525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.795690 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.796247 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.796257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.796259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.796877 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:27.797163 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:27.797559 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 11:12:27.797783 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":420,"threads_started":1}
I20260925 11:12:27.801649 31478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:27.803081 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.802560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56308)
0925 11:12:27.802785 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.802918 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 11:12:27.803385 31427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:27.805100 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.804736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56314)
0925 11:12:27.804891 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.804983 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
W20260925 11:12:27.805465 31478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:27.806874 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:41981
--webserver_interface=127.8.181.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:12:27.854863 31478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 1
I20260925 11:12:27.856112 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.855654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56330)
0925 11:12:27.855851 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.855963 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 11:12:27.857184 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.856853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56338)
0925 11:12:27.856984 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.857068 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 11:12:27.857482 31478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:27.904078 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } attempt: 5
Sep 25 11:12:27 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:27.921048 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.904949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37316)
0925 11:12:27.905189 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.905211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:27.905366 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:27.905841 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:27.905845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:27.906250 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:27.906426 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.906436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.908064 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.908069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.909015 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:27.909030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.910133 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.910138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:27.910301 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.911109 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:27.911129 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:27.913253 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:27.917896 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.917901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.917904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.918461 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.919144 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:27.919149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:27.919151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:27.919318 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:27.920155 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:27.920160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:27.920337 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 11:12:27.920491 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 11:12:27.920753 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 11:12:27.921231 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.905184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.8.181.61:38693)
0925 11:12:27.905399 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:27.905406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:27.905429 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:12:27.905497 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:27.905503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:27.905616 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:27.905733 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:27.906689 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.907809 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.909220 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:27.909876 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:27.910023 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:27.913519 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:27.913527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:27.913531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:27.913575 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:27.917685 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.918698 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.918704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.918707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.918893 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:27.919464 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:27.919468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:27.919472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:27.919938 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:27.920104 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:27.920662 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 11:12:27.920917 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 11:12:27.922735 31404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:27.924075 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.923552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56352)
0925 11:12:27.923769 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.923876 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 11:12:27.925231 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.924824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56362)
0925 11:12:27.924991 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.925100 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 11:12:27.925477 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:27.925689 31478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 2
I20260925 11:12:27.926708 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.926268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56378)
0925 11:12:27.926432 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.926570 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 11:12:27.927655 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.927320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56390)
0925 11:12:27.927467 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.927546 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 11:12:27.927887 31478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:27.976783 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 1
I20260925 11:12:27.978127 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.977556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56406)
0925 11:12:27.977808 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.977918 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 11:12:27.979331 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:27.978894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56418)
0925 11:12:27.979098 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:27.979191 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 11:12:27.979647 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:28.019150 31478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 3
I20260925 11:12:28.020763 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.020116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56426)
0925 11:12:28.020418 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.020544 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 11:12:28.022094 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.021537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56432)
0925 11:12:28.021831 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.021934 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260925 11:12:28.022432 31478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:28.047935 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 2
I20260925 11:12:28.049301 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.048699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56434)
0925 11:12:28.048973 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.049088 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 11:12:28.050514 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.049983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56438)
0925 11:12:28.050190 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.050299 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 11:12:28.050803 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:28.157058 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 3
I20260925 11:12:28.158320 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.157785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56450)
0925 11:12:28.157996 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.158149 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 11:12:28.159391 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.159003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56460)
0925 11:12:28.159218 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.159283 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 11:12:28.159585 31404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
I20260925 11:12:28.165733 31478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 4
I20260925 11:12:28.167052 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.166398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56466)
0925 11:12:28.166681 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.166803 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 11:12:28.168217 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.167834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56476)
0925 11:12:28.167984 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.168081 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 11:12:28.168460 31478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:41981: Network error: Client connection negotiation failed: client connection to 127.8.181.60:41981: connect: Connection refused (error 111)
W20260925 11:12:28.175304 31486 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:28.175707 31486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:28.175832 31486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:28.175967 31486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:28.183702 31486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:28.183846 31486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:28.183970 31486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:28.184055 31486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:28.184161 31486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:28.195261 31486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:44613,127.8.181.61:38693,127.8.181.60:41981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.181.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:28.196658 31486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:28.198210 31486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:28.211035 31491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:28.211117 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:28.211140 31494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:28.212422 31486 server_base.cc:1034] running on GCE node
I20260925 11:12:28.213385 31486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:28.215011 31486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:28.216461 31486 hybrid_clock.cc:648] HybridClock initialized: now 1790334748216219 us; error 66 us; skew 500 ppm
I20260925 11:12:28.216866 31486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334748, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:28.227108 31486 init.cc:377] Logged in from keytab as kudu/127.8.181.60@KRBTEST.COM (short username kudu)
I20260925 11:12:28.228965 31486 webserver.cc:460] Webserver started at http://127.8.181.60:34819/ using document root <none> and password file <none>
I20260925 11:12:28.229781 31486 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:28.229933 31486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:28.230302 31486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:28.233722 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data/instance:
uuid: "243dee745b0b413fb4db621cfaa86b91"
format_stamp: "Formatted at 2026-09-25 11:12:28 on dist-test-slave-6042"
server_key: "b52764fddb13fa3464703a290227d54e"
server_key_iv: "b741f1c2c50973dd604129df57fd04aa"
server_key_version: "encryptionkey@0"
I20260925 11:12:28.234658 31486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal/instance:
uuid: "243dee745b0b413fb4db621cfaa86b91"
format_stamp: "Formatted at 2026-09-25 11:12:28 on dist-test-slave-6042"
server_key: "b52764fddb13fa3464703a290227d54e"
server_key_iv: "b741f1c2c50973dd604129df57fd04aa"
server_key_version: "encryptionkey@0"
I20260925 11:12:28.240295 31486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:12:28.244344 31501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:28.245988 31486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:12:28.246222 31486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "243dee745b0b413fb4db621cfaa86b91"
format_stamp: "Formatted at 2026-09-25 11:12:28 on dist-test-slave-6042"
server_key: "b52764fddb13fa3464703a290227d54e"
server_key_iv: "b741f1c2c50973dd604129df57fd04aa"
server_key_version: "encryptionkey@0"
I20260925 11:12:28.246479 31486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:28.262434 31486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:28.267131 31486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:28.267426 31486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:28.287611 31486 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.60:41981
I20260925 11:12:28.287628 31552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.60:41981 every 8 connection(s)
I20260925 11:12:28.289151 31486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
I20260925 11:12:28.291774 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31486
I20260925 11:12:28.291987 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790334376327139-8916-0/minicluster-data/master-2/wal/instance
I20260925 11:12:28.292544 8916 external_mini_cluster.cc:1428] Setting key 9f0d4ed7f139d01e4e5a1003280dff64
I20260925 11:12:28.293867 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:28.304045 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334748, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:28.318131 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.294623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.8.181.60:41981)
0925 11:12:28.295431 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.295442 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.295485 (+ 43us) server_negotiation.cc:407] Connection header received
0925 11:12:28.297099 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.297122 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.297579 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.298031 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.300061 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.301492 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.303464 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.304252 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.304419 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.309739 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.309750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.309760 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.309806 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.313027 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.313815 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.313822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.313829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.314072 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.314731 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.314735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.314737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.315400 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.315659 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.316382 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.316896 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":394,"threads_started":1}
I20260925 11:12:28.324515 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.307679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:44613 (local address 127.0.0.1:35280)
0925 11:12:28.308682 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.308731 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:28.308921 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:28.309337 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:28.309345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:28.310055 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:28.310404 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.310414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.311550 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.311556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.312711 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.312727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.313497 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.313503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.313697 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.314380 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:28.314405 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:28.317319 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:28.321076 (+ 3757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.321086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.321094 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.321788 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.322289 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.322293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.322295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.322438 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.323235 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:28.323244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:28.323477 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 11:12:28.323750 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 11:12:28.324107 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":584,"threads_started":1}
I20260925 11:12:28.324517 31557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.307718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.8.181.62:44613)
0925 11:12:28.308271 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.308280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.308826 (+ 546us) server_negotiation.cc:407] Connection header received
0925 11:12:28.309073 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.309078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.309145 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.309272 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.310589 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.311378 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.312892 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.313297 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.313456 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.317576 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.317582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.317586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.317634 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.320908 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.321922 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.321931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.321937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.322154 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.322622 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.322632 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.322638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.323089 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.323204 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.323863 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.324031 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260925 11:12:28.326131 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334748, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:28.336812 31404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 4
I20260925 11:12:28.339322 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.321689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.8.181.62:44613)
0925 11:12:28.322365 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.322369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.322385 (+ 16us) server_negotiation.cc:407] Connection header received
0925 11:12:28.322421 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.322425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.322595 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.322706 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.324393 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.325503 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.327000 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.327523 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.327677 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.332016 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.332023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.332028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.332073 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.335143 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.336111 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.336115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.336118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.336249 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.336818 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.336823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.336828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.337243 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.337610 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.338602 (+ 992us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.338884 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":543,"thread_start_us":367,"threads_started":1}
I20260925 11:12:28.342338 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.327049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38693 (local address 127.0.0.1:37332)
0925 11:12:28.327344 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.327360 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:28.327495 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:28.327821 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:28.327826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:28.328222 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:28.328367 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.328375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.329331 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.329337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.330220 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.330231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.331058 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.331063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.331243 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.331892 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:28.331916 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:28.334380 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:28.338310 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.338315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.338318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.338924 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.340236 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.340240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.340242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.340462 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.341227 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:28.341232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:28.341431 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 11:12:28.341641 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 11:12:28.341956 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 11:12:28.342413 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.327065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.8.181.61:38693)
0925 11:12:28.327246 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.327250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.327431 (+ 181us) server_negotiation.cc:407] Connection header received
0925 11:12:28.327614 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.327619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.327696 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.327803 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.328526 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.329188 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.330366 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.330888 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.331043 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.334650 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.334656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.334660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.334695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.338116 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.339063 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.339068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.339072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.339238 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.340601 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.340606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.340610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.340986 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.341159 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.341804 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.342032 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:12:28.347705 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:28.354579 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.337957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.8.181.60:41981)
0925 11:12:28.338168 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.338174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.338191 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:12:28.338246 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.338253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.338329 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.338469 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.339373 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.340119 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.341901 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.342784 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.342938 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.346740 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.346748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.346754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.346793 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.350621 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.351992 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.352001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.352005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.352195 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.352711 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.352719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.352726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.353141 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.353392 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.353908 (+ 516us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.354179 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:12:28.354921 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.337581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56488)
0925 11:12:28.338000 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.338020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:28.338121 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:28.338543 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:28.338548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:28.338964 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:28.339203 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.339216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.340253 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.340261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.341676 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.341697 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.342953 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.342959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.343184 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.343949 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:28.343974 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:28.346427 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:28.350805 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.350814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.351050 (+ 236us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.351834 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.352338 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.352345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.352351 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.352585 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.353297 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:28.353306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:28.353584 (+ 278us) client_negotiation.cc:769] Sending connection context
0925 11:12:28.353791 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 11:12:28.354252 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260925 11:12:28.361254 31404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: Bootstrap starting.
I20260925 11:12:28.362695 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.341191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.8.181.61:38693)
0925 11:12:28.342854 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.342861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.342880 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:12:28.342927 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.342932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.343002 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.343109 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.345008 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.346414 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.348236 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.348996 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.349136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.354694 (+ 5558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.354701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.354709 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.354761 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.358548 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.359455 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.359461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.359464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.359682 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.360166 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.360171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.360173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.360535 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.360824 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.362070 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.362299 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560,"thread_start_us":5477,"threads_started":1}
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334748, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:12:28.366186 31404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:28.367726 31404 log.cc:824] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:28.368356 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:28.372722 31404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: No bootstrap required, opened a new log
I20260925 11:12:28.374891 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.348501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56504)
0925 11:12:28.348934 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.348958 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:28.349068 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:28.356957 (+ 7889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:28.356962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:28.357459 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:28.357655 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.357668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.359162 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.359167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.360483 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.360494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.361242 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.361248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.361446 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.362124 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:28.362146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:28.367059 (+ 4913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:28.372230 (+ 5171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.372237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.372241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.372630 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.373022 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.373027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.373030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.373156 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.373739 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:28.373743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:28.373884 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 11:12:28.374054 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 11:12:28.374446 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":26}
I20260925 11:12:28.374958 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.348492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.8.181.60:41981)
0925 11:12:28.356467 (+ 7975us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.356474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.356495 (+ 21us) server_negotiation.cc:407] Connection header received
0925 11:12:28.356553 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.356559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.356640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.356746 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.357809 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.359003 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.360646 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.361072 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.361225 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.367344 (+ 6119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.367353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.367360 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.367411 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.372088 (+ 4677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.372762 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.372769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.372773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.372902 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.373262 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.373270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.373273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.373612 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.373772 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.374151 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.374605 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":6815,"thread_start_us":20930,"threads_started":1}
I20260925 11:12:28.372298 31379 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5a81cd8e793747398d25420af082e983"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35292
I20260925 11:12:28.379722 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: Bootstrap starting.
I20260925 11:12:28.382296 31404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.382767 31404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a81cd8e793747398d25420af082e983, State: Initialized, Role: FOLLOWER
I20260925 11:12:28.383133 31553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:28.383287 31404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.384500 31553 log.cc:824] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:28.384317 31563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.384814 31563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.385614 31404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:28.385694 31379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 11:12:28.385932 31379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:28.387988 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: No bootstrap required, opened a new log
I20260925 11:12:28.390784 31379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.394215 31379 leader_election.cc:288] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee75bf2f52c45debb92c220c3d11726 (127.8.181.61:38693), 243dee745b0b413fb4db621cfaa86b91 (127.8.181.60:41981)
I20260925 11:12:28.401321 31553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.401907 31553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243dee745b0b413fb4db621cfaa86b91, State: Initialized, Role: FOLLOWER
I20260925 11:12:28.402602 31553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
W20260925 11:12:28.413360 31575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:12:28.413623 31575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:12:28.414734 31579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.415359 31579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.416664 31528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a81cd8e793747398d25420af082e983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "243dee745b0b413fb4db621cfaa86b91"
I20260925 11:12:28.417354 31528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [term 0 FOLLOWER]: Advancing to term 1
W20260925 11:12:28.420718 31358 tablet.cc:2369] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:12:28.401324 31337 master_runner.cc:186] Error getting master registration for 127.8.181.61:38693: OK, instance_id { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" instance_seqno: 1790334747722933 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:12:28.422454 31337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:12:28.422518 31528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a81cd8e793747398d25420af082e983 in term 1.
I20260925 11:12:28.416643 31553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:28.423339 31353 leader_election.cc:302] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243dee745b0b413fb4db621cfaa86b91, 5a81cd8e793747398d25420af082e983; no voters:
I20260925 11:12:28.423804 31563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:28.424554 31563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [term 1 LEADER]: Becoming Leader. State: Replica: 5a81cd8e793747398d25420af082e983, State: Running, Role: LEADER
I20260925 11:12:28.425138 31563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.428913 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a81cd8e793747398d25420af082e983. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.429333 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:28.460669 31586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:28.463984 31586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 11:12:28.467677 31478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } attempt: 5
I20260925 11:12:28.472415 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.434930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.8.181.62:44613)
0925 11:12:28.437671 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.437681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.437728 (+ 47us) server_negotiation.cc:407] Connection header received
0925 11:12:28.437862 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.437872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.438042 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.438226 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.442262 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.443551 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.451056 (+ 7505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.452099 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.452278 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.459953 (+ 7675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.459965 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.459972 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.460042 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.466149 (+ 6107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.467810 (+ 1661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.467819 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.467825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.468144 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.468949 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.468957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.468960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.469643 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.470164 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.471304 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.471759 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 11:12:28 dist-test-slave-6042 krb5kdc[31324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334747, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:12:28.484666 31528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [term 1 FOLLOWER]: Refusing update from remote peer 5a81cd8e793747398d25420af082e983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:12:28.485966 31563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:12:28.491382 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.468731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.8.181.60:41981)
0925 11:12:28.468975 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.468990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.469022 (+ 32us) server_negotiation.cc:407] Connection header received
0925 11:12:28.469092 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.469101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.469202 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.469362 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.470408 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.471687 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.473485 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.474177 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.474310 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.479076 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.479083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.479088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.479134 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.483398 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.488138 (+ 4740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.488145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.488151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.488342 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.488818 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.488826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.488833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.489228 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.489419 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.490652 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.490924 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 11:12:28.491688 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.468349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:41981 (local address 127.0.0.1:56508)
0925 11:12:28.468572 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:28.468594 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:28.468710 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:28.469384 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:28.469389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:28.469964 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:28.470205 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.470219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.471856 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.471861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.473286 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:28.473304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.474529 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.474534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:28.474774 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.475525 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:28.475553 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:28.478833 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:28.483598 (+ 4765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.483605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.483611 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.487991 (+ 4380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.488491 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:28.488497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:28.488501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:28.488678 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:28.489474 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:28.489478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:28.489664 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 11:12:28.489815 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 11:12:28.490144 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 11:12:28.493849 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.436678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.8.181.61:38693)
0925 11:12:28.436937 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.436944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.473972 (+ 37028us) server_negotiation.cc:407] Connection header received
0925 11:12:28.474317 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.474323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.474406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.474499 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.476507 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.477722 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.479573 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.480313 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.480430 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.483831 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.483838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.483842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.483886 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.487767 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.489465 (+ 1698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.489470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.489473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.489745 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.490549 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.490555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.490580 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.491163 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.491753 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.492799 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.493203 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 11:12:28.498113 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: Bootstrap starting.
I20260925 11:12:28.502931 31478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:28.504329 31478 log.cc:824] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:28.507161 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.435124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.8.181.60:41981)
0925 11:12:28.435325 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.435333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.438445 (+ 3112us) server_negotiation.cc:407] Connection header received
0925 11:12:28.439537 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.439548 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.439671 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.439783 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.444546 (+ 4763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.446011 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.448538 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.449394 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.449533 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.461778 (+ 12245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:28.461787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:28.461792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:28.461846 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:28.465684 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.467349 (+ 1665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.467357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.467363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.467547 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:28.468371 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:28.468376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:28.468380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:28.469267 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:28.478084 (+ 8817us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.506424 (+ 28340us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.506758 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:12:28.509610 31478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: No bootstrap required, opened a new log
I20260925 11:12:28.516934 31579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a81cd8e793747398d25420af082e983. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.517293 31579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 11:12:28.519546 31507 tablet.cc:2369] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:12:28.522795 31594 mvcc.cc:205] Tried to move back new op lower bound from 7333211129777049600 to 7333211129554923520. Current Snapshot: MvccSnapshot[applied={T|T < 7333211129777049600}]
I20260925 11:12:28.526127 31602 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 11:12:28.533921 31478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.534720 31478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee75bf2f52c45debb92c220c3d11726, State: Initialized, Role: FOLLOWER
I20260925 11:12:28.535650 31478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } }
I20260925 11:12:28.543248 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.546258 31453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a81cd8e793747398d25420af082e983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ee75bf2f52c45debb92c220c3d11726"
I20260925 11:12:28.546821 31452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:28.546211 31478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:28.549429 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.551167 31452 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [term 1 FOLLOWER]: Refusing update from remote peer 5a81cd8e793747398d25420af082e983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:12:28.552409 31563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:12:28.556064 31453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a81cd8e793747398d25420af082e983 in term 1.
I20260925 11:12:28.556316 31563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.556684 31563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:28.583343 31586 catalog_manager.cc:1347] Generated new cluster ID: b2e3b433439d4e00ba61e8593d55de9e
I20260925 11:12:28.583498 31586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:28.586365 31579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.586747 31579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.568058 31486 master_runner.cc:186] Error getting master registration for 127.8.181.61:38693: OK, instance_id { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" instance_seqno: 1790334747722933 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:12:28.597108 31486 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:12:28.602813 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a81cd8e793747398d25420af082e983. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.603221 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.604104 31610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.604483 31610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:28.604468 31581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a81cd8e793747398d25420af082e983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a81cd8e793747398d25420af082e983" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 44613 } } peers { permanent_uuid: "8ee75bf2f52c45debb92c220c3d11726" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38693 } } peers { permanent_uuid: "243dee745b0b413fb4db621cfaa86b91" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 41981 } } }
I20260925 11:12:28.604813 31581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983 [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:28.610615 31586 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:28.612629 31586 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:28.634883 31586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a81cd8e793747398d25420af082e983: Generated new TSK 0
I20260925 11:12:28.636045 31586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 11:12:28.637676 31621 catalog_manager.cc:1259] Loaded cluster ID: b2e3b433439d4e00ba61e8593d55de9e
I20260925 11:12:28.637825 31621 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: loading cluster ID for follower catalog manager: success
I20260925 11:12:28.643440 31621 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: acquiring CA information for follower catalog manager: success
I20260925 11:12:28.662602 31621 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ee75bf2f52c45debb92c220c3d11726: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 11:12:28.714566 31484 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.698795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.8.181.61:38693)
0925 11:12:28.699193 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.699201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.711793 (+ 12592us) server_negotiation.cc:407] Connection header received
0925 11:12:28.712417 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.712425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.712546 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.712639 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.714175 (+ 1536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37360: BlockingRecv error: recv got EOF from 127.0.0.1:37360 (error 108)
Metrics: {"server-negotiator.queue_time_us":216}
W20260925 11:12:28.718189 31559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.698938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.8.181.62:44613)
0925 11:12:28.699397 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.699406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.715940 (+ 16534us) server_negotiation.cc:407] Connection header received
0925 11:12:28.716253 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.716262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.716387 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.716478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.717797 (+ 1319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35314: BlockingRecv error: recv got EOF from 127.0.0.1:35314 (error 108)
Metrics: {"server-negotiator.queue_time_us":296}
W20260925 11:12:28.721674 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.707416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.8.181.60:41981)
0925 11:12:28.707685 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.707694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.719243 (+ 11549us) server_negotiation.cc:407] Connection header received
0925 11:12:28.720067 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.720078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.720190 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.720276 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.721430 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56516: BlockingRecv error: recv got EOF from 127.0.0.1:56516 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 11:12:28.736582 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.727079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.8.181.61:38693)
0925 11:12:28.727291 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.727297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.728037 (+ 740us) server_negotiation.cc:407] Connection header received
0925 11:12:28.728393 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.728401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.728605 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.728746 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.729468 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.730659 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.733126 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.733933 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.734045 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.734909 (+ 864us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.735697 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.735819 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.736071 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.736183 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 11:12:28.777160 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.774736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.8.181.60:41981)
0925 11:12:28.774967 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.774976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.775558 (+ 582us) server_negotiation.cc:407] Connection header received
0925 11:12:28.775844 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.775849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.775953 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.776084 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.776903 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56532: BlockingRecv error: recv got EOF from 127.0.0.1:56532 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 11:12:28.788767 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.782035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.8.181.61:38693)
0925 11:12:28.782330 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.782340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.782943 (+ 603us) server_negotiation.cc:407] Connection header received
0925 11:12:28.783218 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.783225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.783307 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.783425 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.783899 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.784744 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.786441 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.787158 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.787325 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.787773 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.787936 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.788066 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.788270 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.788356 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 11:12:28.799619 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.796957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.8.181.60:41981)
0925 11:12:28.797227 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.797233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.797256 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:12:28.797338 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.797343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.797431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.797571 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.798399 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56542: BlockingRecv error: recv got EOF from 127.0.0.1:56542 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 11:12:28.810854 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.804511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.8.181.61:38693)
0925 11:12:28.804798 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.804808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.805415 (+ 607us) server_negotiation.cc:407] Connection header received
0925 11:12:28.805648 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.805654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.805743 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.805862 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.806294 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.807256 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.808763 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.809311 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.809474 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.809901 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.810057 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.810175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.810333 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.810443 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 11:12:28.820577 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.818708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.8.181.60:41981)
0925 11:12:28.818911 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.818918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.818942 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:12:28.819172 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.819185 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.819270 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.819337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.820330 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56558: BlockingRecv error: recv got EOF from 127.0.0.1:56558 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 11:12:28.835294 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.828457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.8.181.61:38693)
0925 11:12:28.828813 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.828824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.829419 (+ 595us) server_negotiation.cc:407] Connection header received
0925 11:12:28.829668 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.829679 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.829755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.829879 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.830318 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.831293 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.832918 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.833634 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.833804 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.834262 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.834432 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.834591 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.834833 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.834932 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
W20260925 11:12:28.844698 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.842038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.8.181.60:41981)
0925 11:12:28.842414 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.842425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.843087 (+ 662us) server_negotiation.cc:407] Connection header received
0925 11:12:28.843333 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.843340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.843435 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.843564 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.844423 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56566: BlockingRecv error: recv got EOF from 127.0.0.1:56566 (error 108)
Metrics: {"server-negotiator.queue_time_us":191}
I20260925 11:12:28.862224 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.856495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.8.181.61:38693)
0925 11:12:28.856735 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.856745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.857302 (+ 557us) server_negotiation.cc:407] Connection header received
0925 11:12:28.857536 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.857542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.857627 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.857736 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.858152 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.858997 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.860354 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.860810 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.860948 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.861371 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.861487 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.861591 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.861773 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.861858 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 11:12:28.871312 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.868940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.8.181.60:41981)
0925 11:12:28.869198 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.869208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.869734 (+ 526us) server_negotiation.cc:407] Connection header received
0925 11:12:28.869949 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.869956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.870055 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.870174 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.871068 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56568: BlockingRecv error: recv got EOF from 127.0.0.1:56568 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 11:12:28.903199 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.896577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.8.181.61:38693)
0925 11:12:28.896851 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.896861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.897469 (+ 608us) server_negotiation.cc:407] Connection header received
0925 11:12:28.897733 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.897740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.897822 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.897941 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.898399 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.899594 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.901118 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.901630 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.901786 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.902279 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.902422 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.902586 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.902748 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.902849 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 11:12:28.913008 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.910302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.8.181.60:41981)
0925 11:12:28.910617 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.910627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.911223 (+ 596us) server_negotiation.cc:407] Connection header received
0925 11:12:28.911482 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.911489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.911608 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.911735 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.912713 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56574: BlockingRecv error: recv got EOF from 127.0.0.1:56574 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 11:12:28.956990 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:28.950045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.8.181.61:38693)
0925 11:12:28.950361 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.950371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.951051 (+ 680us) server_negotiation.cc:407] Connection header received
0925 11:12:28.951371 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.951377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.951515 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.951642 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:28.952128 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.952968 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.954864 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:28.955394 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:28.955552 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:28.956054 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:28.956182 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:28.956295 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:28.956472 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 11:12:28.956561 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 11:12:28.965981 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:28.964094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.8.181.60:41981)
0925 11:12:28.964311 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:28.964317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:28.964339 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:12:28.964583 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:28.964590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:28.964684 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:28.964750 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:28.965736 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56576: BlockingRecv error: recv got EOF from 127.0.0.1:56576 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 11:12:29.039749 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:29.033801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.8.181.61:38693)
0925 11:12:29.034062 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.034072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.034667 (+ 595us) server_negotiation.cc:407] Connection header received
0925 11:12:29.034908 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.034914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.035078 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.035200 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:29.035668 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.036454 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.037878 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.038335 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.038486 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:29.038928 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:29.039067 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:29.039172 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:29.039324 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 11:12:29.039412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260925 11:12:29.048334 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:29.045878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.8.181.60:41981)
0925 11:12:29.046140 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.046149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.046772 (+ 623us) server_negotiation.cc:407] Connection header received
0925 11:12:29.046994 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.047002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.047091 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.047227 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:29.048102 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56592: BlockingRecv error: recv got EOF from 127.0.0.1:56592 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 11:12:29.185606 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:29.180227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.8.181.61:38693)
0925 11:12:29.180490 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.180498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.180754 (+ 256us) server_negotiation.cc:407] Connection header received
0925 11:12:29.180994 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.181018 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.181090 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.181203 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:29.181664 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.182388 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.183823 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.184241 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.184390 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:29.184817 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:29.184955 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:29.185069 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:29.185210 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 11:12:29.185294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 11:12:29.195670 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:29.193311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.8.181.60:41981)
0925 11:12:29.193621 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.193631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.194204 (+ 573us) server_negotiation.cc:407] Connection header received
0925 11:12:29.194365 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.194372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.194514 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.194688 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:29.195419 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 11:12:29.461155 31484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:29.455812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.8.181.61:38693)
0925 11:12:29.456076 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.456083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.456307 (+ 224us) server_negotiation.cc:407] Connection header received
0925 11:12:29.456586 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.456590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.456669 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.456777 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:29.457197 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.457967 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.459467 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.459874 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.460021 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:29.460427 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:29.460538 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:29.460649 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:29.460794 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 11:12:29.460874 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260925 11:12:29.469692 31556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:29.467683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.8.181.60:41981)
0925 11:12:29.467950 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.467955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.468145 (+ 190us) server_negotiation.cc:407] Connection header received
0925 11:12:29.468323 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.468328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.468406 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.468505 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:29.469385 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56604: BlockingRecv error: recv got EOF from 127.0.0.1:56604 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 11:12:29.530753 31602 catalog_manager.cc:1259] Loaded cluster ID: b2e3b433439d4e00ba61e8593d55de9e
I20260925 11:12:29.530936 31602 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: loading cluster ID for follower catalog manager: success
I20260925 11:12:29.533874 31602 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: acquiring CA information for follower catalog manager: success
I20260925 11:12:29.535943 31602 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 243dee745b0b413fb4db621cfaa86b91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:12:29.991711 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:29.985117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.8.181.61:38693)
0925 11:12:29.985703 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.985710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.985867 (+ 157us) server_negotiation.cc:407] Connection header received
0925 11:12:29.986237 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.986245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.986335 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.986461 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:29.986977 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.987847 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.989532 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:29.990086 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:29.990240 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:29.990790 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:29.990902 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:29.991031 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:29.991236 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 11:12:29.991325 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
I20260925 11:12:30.004837 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:29.998413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.8.181.60:41981)
0925 11:12:29.999033 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:29.999038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:29.999058 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:12:29.999217 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:29.999221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:29.999401 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:29.999517 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:30.000001 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.000847 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.002517 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.002941 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.003050 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:30.003624 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:30.004280 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:30.004391 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:30.004526 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 11:12:30.004602 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":368,"threads_started":1}
I20260925 11:12:30.023526 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:30.012610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.8.181.60:41981)
0925 11:12:30.013131 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:30.013140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:30.013894 (+ 754us) server_negotiation.cc:407] Connection header received
0925 11:12:30.014683 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:30.014694 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:30.014831 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:30.014918 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:30.016178 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.017747 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.021149 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.021794 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.021890 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:30.022688 (+ 798us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:30.022816 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:30.022938 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:30.023109 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 11:12:30.023198 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260925 11:12:30.026710 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:30.012641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.8.181.61:38693)
0925 11:12:30.012894 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:30.012903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:30.019681 (+ 6778us) server_negotiation.cc:407] Connection header received
0925 11:12:30.019935 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:30.019945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:30.020098 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:30.020207 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:30.020920 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.021996 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.024570 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.025203 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.025380 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:30.025879 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:30.026004 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:30.026121 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:30.026279 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 11:12:30.026372 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 11:12:30.028656 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:30.012173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.8.181.62:44613)
0925 11:12:30.020677 (+ 8504us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:30.020684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:30.021179 (+ 495us) server_negotiation.cc:407] Connection header received
0925 11:12:30.021387 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:30.021391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:30.021574 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:30.021784 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:30.023030 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.024576 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.026243 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:30.026817 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:30.026973 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:30.027526 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:30.028031 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:30.028151 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:30.028263 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 11:12:30.028335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8359,"thread_start_us":8230,"threads_started":1}
I20260925 11:12:30.034802 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31337
I20260925 11:12:30.036492 31399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:30.150382 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31337
W20260925 11:12:30.171309 31505 connection.cc:535] server connection from 127.0.0.1:56488 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 11:12:30.171697 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31411
I20260925 11:12:30.172827 31473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:30.282243 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31411
I20260925 11:12:30.309540 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31486
I20260925 11:12:30.310807 31548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:30.419373 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31486
2026-09-25T11:12:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5130 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 11:12:30 dist-test-slave-6042 krb5kdc[31758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 11:12:30 dist-test-slave-6042 krb5kdc[31758](info): set up 2 sockets
Sep 25 11:12:30 dist-test-slave-6042 krb5kdc[31758](info): commencing operation
krb5kdc: starting...
W20260925 11:12:31.559736 8916 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/test-user.keytab.
Sep 25 11:12:31 dist-test-slave-6042 krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T11:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T11:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.181.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
Entry for principal kudu/127.8.181.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab.
I20260925 11:12:31.856149 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39329
--webserver_interface=127.8.181.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 11:12:32.202584 31771 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:32.202983 31771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:32.203122 31771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:32.203248 31771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:32.211162 31771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:32.211277 31771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:32.211336 31771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:32.211393 31771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:32.211458 31771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:32.221936 31771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.62.keytab
--principal=kudu/127.8.181.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.62:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.181.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:32.223464 31771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:32.224833 31771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:32.237982 31776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:32.237916 31777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:32.237958 31779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:32.238575 31771 server_base.cc:1034] running on GCE node
I20260925 11:12:32.239586 31771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:32.241232 31771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:32.242626 31771 hybrid_clock.cc:648] HybridClock initialized: now 1790334752242506 us; error 71 us; skew 500 ppm
I20260925 11:12:32.243031 31771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:32 dist-test-slave-6042 krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:32.246800 31771 init.cc:377] Logged in from keytab as kudu/127.8.181.62@KRBTEST.COM (short username kudu)
I20260925 11:12:32.248555 31771 webserver.cc:460] Webserver started at http://127.8.181.62:35863/ using document root <none> and password file <none>
I20260925 11:12:32.249330 31771 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:32.249486 31771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:32.249846 31771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:32.253310 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data/instance:
uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "3ffe1590df11f352e1513c005be6e15b"
server_key_iv: "0031813d02943cf5d083db0380cc5953"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.254339 31771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance:
uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "3ffe1590df11f352e1513c005be6e15b"
server_key_iv: "0031813d02943cf5d083db0380cc5953"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.260131 31771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 11:12:32.264081 31786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:32.265841 31771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 11:12:32.266023 31771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal
uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "3ffe1590df11f352e1513c005be6e15b"
server_key_iv: "0031813d02943cf5d083db0380cc5953"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.266227 31771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:32.287671 31771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:32.292292 31771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:32.292543 31771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:32.313225 31771 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.62:39329
I20260925 11:12:32.313272 31837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.62:39329 every 8 connection(s)
I20260925 11:12:32.314729 31771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/data/info.pb
I20260925 11:12:32.315605 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31771
I20260925 11:12:32.315815 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-0/wal/instance
I20260925 11:12:32.316493 8916 external_mini_cluster.cc:1428] Setting key 15d43fbaf53bd978cb7b162a71cccb71
I20260925 11:12:32.319305 31838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:32.327454 31838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.181.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.61@KRBTEST.COM" created.
Sep 25 11:12:32 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:32.357615 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.331932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:39329 (local address 127.0.0.1:41454)
0925 11:12:32.332945 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.333000 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:32.336044 (+ 3044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:32.337245 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:32.337257 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:32.338174 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:32.338757 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.338798 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.340700 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.340708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.342888 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.342910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.344053 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.344065 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.344295 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.345068 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:32.345093 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:32.349487 (+ 4394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:32.353027 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.353040 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.353054 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.353493 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.353939 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.353946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.353952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.354112 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.355027 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:32.355035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:32.355269 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 11:12:32.355520 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 11:12:32.356207 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":169,"thread_start_us":421,"threads_started":1}
I20260925 11:12:32.357667 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.331932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.8.181.62:39329)
0925 11:12:32.334094 (+ 2162us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:32.334115 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:32.334161 (+ 46us) server_negotiation.cc:407] Connection header received
0925 11:12:32.336563 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:32.336570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:32.336885 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:32.337062 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:32.339047 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.340491 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.343107 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.343743 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.343903 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.349787 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:32.349796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:32.349810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:32.349842 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:32.352856 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.353661 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.353668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.353670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.353820 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.354237 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.354241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.354245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.354878 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:32.355107 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:32.355675 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 11:12:32.356031 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1936,"spinlock_wait_cycles":16256,"thread_start_us":1670,"threads_started":1}
I20260925 11:12:32.361166 31838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:32.362659 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.361969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33238)
0925 11:12:32.362228 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.362342 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260925 11:12:32.362921 31787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
I20260925 11:12:32.364981 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.364386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33242)
0925 11:12:32.364618 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.364733 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 11:12:32.365267 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
Entry for principal kudu/127.8.181.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab.
I20260925 11:12:32.388033 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:38125
--webserver_interface=127.8.181.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:12:32.414659 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } attempt: 1
I20260925 11:12:32.415943 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.415440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33246)
0925 11:12:32.415679 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.415801 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 11:12:32.417279 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.416778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33252)
0925 11:12:32.416935 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.417016 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 11:12:32.417582 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
I20260925 11:12:32.485918 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } attempt: 2
I20260925 11:12:32.487197 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.486709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33262)
0925 11:12:32.486949 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.487045 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 11:12:32.488286 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.487927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33274)
0925 11:12:32.488071 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.488158 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 11:12:32.488534 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
I20260925 11:12:32.579849 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } attempt: 3
I20260925 11:12:32.581353 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.580655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33286)
0925 11:12:32.580936 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.581106 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 11:12:32.582693 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.582166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33298)
0925 11:12:32.582352 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.582435 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 11:12:32.582950 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
I20260925 11:12:32.726290 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } attempt: 4
I20260925 11:12:32.727620 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.727073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33310)
0925 11:12:32.727401 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.727488 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 11:12:32.728726 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33320)
0925 11:12:32.728527 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.728608 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 11:12:32.728971 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.61:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.61:38125: connect: Connection refused (error 111)
W20260925 11:12:32.757475 31845 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:32.757969 31845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:32.758293 31845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:32.758487 31845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:32.766542 31845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:32.766662 31845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:32.766764 31845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:32.766837 31845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:32.766945 31845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:32.777438 31845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.61.keytab
--principal=kudu/127.8.181.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.61:38125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.181.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:32.778868 31845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:32.780311 31845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:32.793565 31853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:32.793589 31850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:32.793615 31851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:32.793977 31845 server_base.cc:1034] running on GCE node
I20260925 11:12:32.795306 31845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:32.796842 31845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:32.798158 31845 hybrid_clock.cc:648] HybridClock initialized: now 1790334752798065 us; error 73 us; skew 500 ppm
I20260925 11:12:32.798609 31845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:32 dist-test-slave-6042 krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:32.802203 31845 init.cc:377] Logged in from keytab as kudu/127.8.181.61@KRBTEST.COM (short username kudu)
I20260925 11:12:32.803937 31845 webserver.cc:460] Webserver started at http://127.8.181.61:37065/ using document root <none> and password file <none>
I20260925 11:12:32.804816 31845 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:32.804944 31845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:32.805264 31845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:32.808493 31845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data/instance:
uuid: "8eae95a4384c438f867b15d305059776"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "0dd02e48b76a0fe27d0b5c9916d1220c"
server_key_iv: "a52ec49aabbfc622e262be7b156896a1"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.809441 31845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal/instance:
uuid: "8eae95a4384c438f867b15d305059776"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "0dd02e48b76a0fe27d0b5c9916d1220c"
server_key_iv: "a52ec49aabbfc622e262be7b156896a1"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.815479 31845 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 11:12:32.819556 31860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:32.821429 31845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 11:12:32.821708 31845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal
uuid: "8eae95a4384c438f867b15d305059776"
format_stamp: "Formatted at 2026-09-25 11:12:32 on dist-test-slave-6042"
server_key: "0dd02e48b76a0fe27d0b5c9916d1220c"
server_key_iv: "a52ec49aabbfc622e262be7b156896a1"
server_key_version: "encryptionkey@0"
I20260925 11:12:32.822002 31845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:32.849337 31845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:32.854106 31845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:32.854437 31845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:32.878553 31845 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.61:38125
I20260925 11:12:32.878590 31911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.61:38125 every 8 connection(s)
I20260925 11:12:32.880079 31845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/data/info.pb
I20260925 11:12:32.884109 31912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:32.888000 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31845
I20260925 11:12:32.888222 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-1/wal/instance
I20260925 11:12:32.888702 8916 external_mini_cluster.cc:1428] Setting key 27fa04629d4025c8572176b33cfb0826
I20260925 11:12:32.892578 31912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:32 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.181.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.181.60@KRBTEST.COM" created.
I20260925 11:12:32.919727 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.895822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.8.181.62:39329)
0925 11:12:32.896824 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:32.896836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:32.897510 (+ 674us) server_negotiation.cc:407] Connection header received
0925 11:12:32.901040 (+ 3530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:32.901047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:32.901187 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:32.901331 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:32.903704 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.904551 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.906289 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.906810 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.906963 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.911146 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:32.911152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:32.911156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:32.911204 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:32.915618 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.916680 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.916686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.916689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.916841 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.917382 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.917387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.917389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.917757 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:32.917960 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:32.918981 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0925 11:12:32.919282 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":416,"threads_started":1}
I20260925 11:12:32.920909 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.896357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:39329 (local address 127.0.0.1:41462)
0925 11:12:32.897311 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.897353 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:32.900518 (+ 3165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:32.901902 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:32.901915 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:32.902705 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:32.903363 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.903384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.904773 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.904777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.906101 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.906116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.907086 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.907095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.907346 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.908135 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:32.908160 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:32.910799 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:32.915866 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.915879 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.915909 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.916484 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.916984 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.916990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.916994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.917225 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.917929 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:32.917940 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:32.918529 (+ 589us) client_negotiation.cc:769] Sending connection context
0925 11:12:32.918863 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 11:12:32.919497 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":420,"threads_started":1}
I20260925 11:12:32.923295 31912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:32 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:32.943845 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.924121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33326)
0925 11:12:32.924408 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.924428 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:32.924554 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:32.925969 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:32.925973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:32.926705 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:32.926931 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.926941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.929147 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.929155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.930766 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:32.930789 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.932004 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.932014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:32.932325 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.933234 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:32.933263 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:32.936349 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:32.940966 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.940973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.940978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.941404 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.941845 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:32.941852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:32.941855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:32.942007 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:32.942762 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:32.942772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:32.942949 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 11:12:32.943100 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 11:12:32.943423 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 11:12:32.943958 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.924465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.8.181.61:38125)
0925 11:12:32.925287 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:32.925298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:32.925335 (+ 37us) server_negotiation.cc:407] Connection header received
0925 11:12:32.925403 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:32.925415 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:32.925717 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:32.925910 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:32.927153 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.928925 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.931007 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:32.931713 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:32.931842 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:32.936643 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:32.936661 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:32.936682 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:32.936720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:32.940767 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.941584 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.941592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.941594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.941722 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:32.942130 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:32.942134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:32.942142 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:32.942609 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:32.942877 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:32.943255 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 11:12:32.943576 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260925 11:12:32.946568 31912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:32.948022 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.947489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40644)
0925 11:12:32.947738 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.947855 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260925 11:12:32.948413 31864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:32.950525 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:32.950032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40652)
0925 11:12:32.950206 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:32.950331 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 11:12:32.950876 31912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
Entry for principal kudu/127.8.181.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab.
I20260925 11:12:32.960737 8916 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
/tmp/dist-test-taskA5RC6A/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:38125
--webserver_interface=127.8.181.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.181.20:45163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 11:12:33.000300 31912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 1
I20260925 11:12:33.001685 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.001109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40668)
0925 11:12:33.001344 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.001466 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 11:12:33.003018 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.002584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40672)
0925 11:12:33.002753 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.002870 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 11:12:33.003288 31912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.028259 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } attempt: 5
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:33.044708 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.029107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33336)
0925 11:12:33.029399 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.029417 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.029526 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.029862 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.029865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.030319 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.030623 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.030635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.032004 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.032008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.033329 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.033346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.034369 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.034374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.034671 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.035365 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.035385 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.037468 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.041574 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.041579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.041581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.042264 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.042877 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.042881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.042884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.043137 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.043868 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.043872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.044031 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.044171 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.044433 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260925 11:12:33.044920 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.029287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.8.181.61:38125)
0925 11:12:33.029513 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.029520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.029543 (+ 23us) server_negotiation.cc:407] Connection header received
0925 11:12:33.029619 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.029624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.029714 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.029830 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.030782 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.031852 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.033505 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.034193 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.034335 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.037720 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.037727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.037732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.037775 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.041382 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.042433 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.042439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.042443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.042675 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.043272 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.043277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.043281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.043713 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.043883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.044306 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.044568 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 11:12:33.046243 31838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:33.047394 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.046969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40686)
0925 11:12:33.047157 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.047251 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 11:12:33.048480 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.048114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40696)
0925 11:12:33.048300 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.048377 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 11:12:33.048740 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.071614 31912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 2
I20260925 11:12:33.073041 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.072472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40698)
0925 11:12:33.072724 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.072837 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 11:12:33.074385 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.073879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40706)
0925 11:12:33.074085 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.074215 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 11:12:33.074906 31912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.100033 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 1
I20260925 11:12:33.101418 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.100881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40714)
0925 11:12:33.101095 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.101245 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 11:12:33.102623 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.102181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40718)
0925 11:12:33.102352 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.102438 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 11:12:33.102860 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.166242 31912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 3
I20260925 11:12:33.167807 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.167160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40728)
0925 11:12:33.167443 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.167576 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 11:12:33.169452 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.168883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40730)
0925 11:12:33.169177 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.169290 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260925 11:12:33.169780 31912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.171211 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 2
I20260925 11:12:33.172362 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.171865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40736)
0925 11:12:33.172110 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.172205 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 11:12:33.173482 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.173112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40742)
0925 11:12:33.173270 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.173364 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 11:12:33.173702 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.279943 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 3
I20260925 11:12:33.281211 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.280716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40748)
0925 11:12:33.280940 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.281030 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 11:12:33.282334 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.281924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40762)
0925 11:12:33.282153 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.282229 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 11:12:33.282562 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.313047 31912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 4
I20260925 11:12:33.314441 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.313858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40778)
0925 11:12:33.314141 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.314267 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 11:12:33.315668 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.315239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40784)
0925 11:12:33.315401 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.315497 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 11:12:33.315935 31912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
W20260925 11:12:33.347178 31920 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 11:12:33.347592 31920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 11:12:33.347718 31920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 11:12:33.347852 31920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 11:12:33.356503 31920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 11:12:33.356695 31920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 11:12:33.356807 31920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 11:12:33.356906 31920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 11:12:33.357024 31920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 11:12:33.367936 31920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.181.20:45163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.181.62:39329,127.8.181.61:38125,127.8.181.60:38125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/krb5kdc/kudu_127.8.181.60.keytab
--principal=kudu/127.8.181.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.181.60:38125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.181.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 09:24:16 UTC on ccycloud.quasar-kczgym.root.comops.site
build id 4672077
ASAN enabled
I20260925 11:12:33.369508 31920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 11:12:33.371047 31920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 11:12:33.384626 31925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:33.384649 31928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 11:12:33.384722 31926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 11:12:33.385535 31920 server_base.cc:1034] running on GCE node
I20260925 11:12:33.386539 31920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 11:12:33.388154 31920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 11:12:33.389428 31920 hybrid_clock.cc:648] HybridClock initialized: now 1790334753389344 us; error 54 us; skew 500 ppm
I20260925 11:12:33.389860 31920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334753, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 11:12:33.399084 31920 init.cc:377] Logged in from keytab as kudu/127.8.181.60@KRBTEST.COM (short username kudu)
I20260925 11:12:33.400954 31920 webserver.cc:460] Webserver started at http://127.8.181.60:35153/ using document root <none> and password file <none>
I20260925 11:12:33.401727 31920 fs_manager.cc:359] Metadata directory not provided
I20260925 11:12:33.401873 31920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 11:12:33.402205 31920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 11:12:33.405617 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data/instance:
uuid: "e929cb01e48d4138a9d531f0796b4292"
format_stamp: "Formatted at 2026-09-25 11:12:33 on dist-test-slave-6042"
server_key: "4115f7b112f965e5a4d91a854165e956"
server_key_iv: "7c44dc5df45a5eafaa59b91bc0f6c25b"
server_key_version: "encryptionkey@0"
I20260925 11:12:33.406807 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal/instance:
uuid: "e929cb01e48d4138a9d531f0796b4292"
format_stamp: "Formatted at 2026-09-25 11:12:33 on dist-test-slave-6042"
server_key: "4115f7b112f965e5a4d91a854165e956"
server_key_iv: "7c44dc5df45a5eafaa59b91bc0f6c25b"
server_key_version: "encryptionkey@0"
I20260925 11:12:33.412555 31920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 11:12:33.416363 31935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 11:12:33.417817 31920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 11:12:33.418038 31920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data,/tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal
uuid: "e929cb01e48d4138a9d531f0796b4292"
format_stamp: "Formatted at 2026-09-25 11:12:33 on dist-test-slave-6042"
server_key: "4115f7b112f965e5a4d91a854165e956"
server_key_iv: "7c44dc5df45a5eafaa59b91bc0f6c25b"
server_key_version: "encryptionkey@0"
I20260925 11:12:33.418318 31920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 11:12:33.437381 31920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 11:12:33.442224 31920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 11:12:33.442663 31920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 11:12:33.459800 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 4
I20260925 11:12:33.460942 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.460497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40786)
0925 11:12:33.460702 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.460801 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 11:12:33.462327 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.461879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40790)
0925 11:12:33.462067 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.462162 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 11:12:33.462678 31838 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.181.60:38125: Network error: Client connection negotiation failed: client connection to 127.8.181.60:38125: connect: Connection refused (error 111)
I20260925 11:12:33.464078 31920 rpc_server.cc:307] RPC server started. Bound to: 127.8.181.60:38125
I20260925 11:12:33.464097 31986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.181.60:38125 every 8 connection(s)
I20260925 11:12:33.465468 31920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/data/info.pb
I20260925 11:12:33.469406 31987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 11:12:33.471696 8916 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu as pid 31920
I20260925 11:12:33.471899 8916 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskA5RC6A/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790334376327139-8916-0/minicluster-data/master-2/wal/instance
I20260925 11:12:33.472470 8916 external_mini_cluster.cc:1428] Setting key 6b3fdd9b38d34fcf8ef330af6b4fc37c
I20260925 11:12:33.479391 31987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334753, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:33.496759 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.474697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.8.181.60:38125)
0925 11:12:33.475493 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.475505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.475549 (+ 44us) server_negotiation.cc:407] Connection header received
0925 11:12:33.477183 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.477207 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.477656 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.478068 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.479634 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.480993 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.482595 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.483553 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.483821 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.488012 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.488025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.488047 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.488116 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.491906 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.492643 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.492659 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.492696 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.492899 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.493305 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.493310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.493313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.494014 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.494355 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.494844 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.495335 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":337,"threads_started":1}
I20260925 11:12:33.498355 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.481909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.62:39329 (local address 127.0.0.1:41472)
0925 11:12:33.483005 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.483047 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.483239 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.483654 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.483661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.484341 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.484586 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.484594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.485614 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.485619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.486717 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.486733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.487644 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.487652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.487809 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.489114 (+ 1305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.489157 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.491910 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.495335 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.495345 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.495351 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.495887 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.496360 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.496364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.496367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.496538 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.497182 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.497190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.497360 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.497543 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.497909 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":759,"threads_started":1}
I20260925 11:12:33.498344 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.481949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.8.181.62:39329)
0925 11:12:33.482445 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.482454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.483152 (+ 698us) server_negotiation.cc:407] Connection header received
0925 11:12:33.483391 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.483395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.483462 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.483582 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.484743 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.485450 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.486897 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.487378 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.487527 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.492130 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.492135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.492138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.492181 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.495186 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.496038 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.496042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.496044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.496237 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.496658 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.496661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.496663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.496981 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.497205 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.497676 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.497905 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
I20260925 11:12:33.499912 31987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334753, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.62@KRBTEST.COM
I20260925 11:12:33.514171 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.500638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.61:38125 (local address 127.0.0.1:33340)
0925 11:12:33.500809 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.500828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.500916 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.501228 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.501233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.501610 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.501756 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.501764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.502947 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.502954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.503875 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.503886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.504578 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.504584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.504766 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.505398 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.505423 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.507510 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.511424 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.511432 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.511438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.511982 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.512414 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.512421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.512427 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.512606 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.513252 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.513256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.513405 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.513521 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.513745 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 11:12:33.514246 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.500738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.8.181.61:38125)
0925 11:12:33.500895 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.500899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.500914 (+ 15us) server_negotiation.cc:407] Connection header received
0925 11:12:33.501025 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.501029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.501086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.501207 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.501900 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.502812 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.504033 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.504433 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.504576 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.507729 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.507735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.507738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.507773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.511256 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.512133 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.512137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.512140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.512281 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.512742 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.512746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.512748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.513078 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.513230 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.513693 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.513916 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 11:12:33.515468 31987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 11:12:33.516134 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.500052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.8.181.62:39329)
0925 11:12:33.500205 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.500211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.500257 (+ 46us) server_negotiation.cc:407] Connection header received
0925 11:12:33.500515 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.500520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.500613 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.500760 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.502381 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.503120 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.504376 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.504799 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.504941 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.509540 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.509545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.509548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.509583 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.512953 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.513855 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.513861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.513865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.514016 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.514514 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.514522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.514526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.514909 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.515227 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.515600 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.515830 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334753, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.60@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.181.61@KRBTEST.COM
I20260925 11:12:33.529816 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.516430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.8.181.60:38125)
0925 11:12:33.516635 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.516641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.516827 (+ 186us) server_negotiation.cc:407] Connection header received
0925 11:12:33.516983 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.516987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.517044 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.517204 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.518188 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.518959 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.520119 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.520522 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.520672 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.524115 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.524120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.524123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.524151 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.527193 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.527858 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.527865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.527869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.527982 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.528362 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.528370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.528373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.528667 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.528828 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.529174 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.529416 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":68}
I20260925 11:12:33.529894 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.516451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40812)
0925 11:12:33.516726 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.516747 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.516872 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.517382 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.517386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.517830 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.518037 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.518051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.519089 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.519097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.519964 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.519975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.520674 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.520679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.520908 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.521571 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.521602 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.523918 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.527342 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.527349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.527354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.527726 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.528097 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.528112 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.528116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.528247 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.528789 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.528794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.528934 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.529108 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.529556 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":149}
I20260925 11:12:33.533967 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.518013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.8.181.61:38125)
0925 11:12:33.518209 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.518214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.518231 (+ 17us) server_negotiation.cc:407] Connection header received
0925 11:12:33.518411 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.518416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.518543 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.518674 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.519860 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.520584 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.522011 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.522526 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.522654 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.527817 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.527823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.527826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.527863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.531149 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.532061 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.532067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.532070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.532216 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.532661 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.532664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.532666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.532923 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.533111 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.533488 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.533688 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 11:12:33.536054 31987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: Bootstrap starting.
I20260925 11:12:33.539723 8916 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 11:12:33.540071 31987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:33.541388 31987 log.cc:824] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:33.544857 31987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: No bootstrap required, opened a new log
I20260925 11:12:33.543016 31813 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478
I20260925 11:12:33.556200 31987 raft_consensus.cc:348] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.556653 31987 raft_consensus.cc:729] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e929cb01e48d4138a9d531f0796b4292, State: Initialized, Role: FOLLOWER
I20260925 11:12:33.557202 31987 consensus_queue.cc:260] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.558166 31994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.558532 31994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:33.559427 31987 sys_catalog.cc:564] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 11:12:33.569274 32005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:12:33.569427 32005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:12:33.567215 31920 master_runner.cc:186] Error getting master registration for 127.8.181.62:39329: OK, instance_id { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" instance_seqno: 1790334752307567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:12:33.578485 31920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 11:12:33.615154 31912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 5
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.61@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:12:33.633539 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.615947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40826)
0925 11:12:33.616376 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.616401 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.616589 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.616974 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.616978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.617434 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.617608 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.617617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.618871 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.618878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.619788 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.619800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.620618 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.620625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.620839 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.621564 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.621587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.624845 (+ 3258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.629506 (+ 4661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.629513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.629519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.630264 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.630964 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.630970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.630977 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.631349 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.632296 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.632302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.632507 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.632701 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.633082 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260925 11:12:33.633560 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.616234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.8.181.60:38125)
0925 11:12:33.616441 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.616449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.616473 (+ 24us) server_negotiation.cc:407] Connection header received
0925 11:12:33.616749 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.616755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.616840 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.616947 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.617786 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.618702 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.619952 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.620443 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.620600 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.625235 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.625244 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.625250 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.625304 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.629233 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.630524 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.630532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.630537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.630782 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.631521 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.631527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.631531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.632113 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.632302 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.632831 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.633120 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 11:12:33.640326 31912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: Bootstrap starting.
I20260925 11:12:33.646404 31912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:33.648298 31912 log.cc:824] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:33.652575 31912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: No bootstrap required, opened a new log
I20260925 11:12:33.667897 31912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.668480 31912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eae95a4384c438f867b15d305059776, State: Initialized, Role: FOLLOWER
I20260925 11:12:33.669229 31912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.670711 32008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.671274 32008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:33.672425 31912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 11:12:33.685030 32019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 11:12:33.685328 32019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 11:12:33.681864 31845 master_runner.cc:186] Error getting master registration for 127.8.181.62:39329: OK, instance_id { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" instance_seqno: 1790334752307567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 11:12:33.695124 31845 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 11:12:33.695395 31941 tablet.cc:2369] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:12:33.739954 31838 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } attempt: 5
Sep 25 11:12:33 dist-test-slave-6042 krb5kdc[31758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790334752, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.181.62@KRBTEST.COM for kudu/127.8.181.60@KRBTEST.COM
I20260925 11:12:33.759347 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.741403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.8.181.60:38125)
0925 11:12:33.741602 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.741608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.741627 (+ 19us) server_negotiation.cc:407] Connection header received
0925 11:12:33.741693 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.741699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.741783 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.741891 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.742971 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.744157 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.746153 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.746881 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.747022 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.751388 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.751395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.751401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.751451 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.755517 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.756480 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.756486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.756490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.756674 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.757216 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.757220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.757224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.757566 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.757714 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.758640 (+ 926us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.758889 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 11:12:33.760438 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.740738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.181.60:38125 (local address 127.0.0.1:40828)
0925 11:12:33.740987 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 11:12:33.741021 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 11:12:33.741158 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 11:12:33.741948 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 11:12:33.741955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 11:12:33.742555 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 11:12:33.742778 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.742791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.744380 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.744386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.745978 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 11:12:33.746006 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.747067 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.747074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 11:12:33.747270 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.748015 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 11:12:33.748040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 11:12:33.751143 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 11:12:33.755729 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.755736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.755740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.756309 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.756822 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 11:12:33.756827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 11:12:33.756831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 11:12:33.757069 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 11:12:33.757799 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 11:12:33.757805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 11:12:33.757972 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 11:12:33.758137 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 11:12:33.758451 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 11:12:33.767750 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: Bootstrap starting.
I20260925 11:12:33.772195 31838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: Neither blocks nor log segments found. Creating new log.
I20260925 11:12:33.773600 31838 log.cc:824] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: Log is configured to *not* fsync() on all Append() calls
I20260925 11:12:33.777498 31838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: No bootstrap required, opened a new log
I20260925 11:12:33.785657 31838 raft_consensus.cc:348] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.786104 31838 raft_consensus.cc:729] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79e2978d8844ce6ab2cb8cbcf6908cc, State: Initialized, Role: FOLLOWER
I20260925 11:12:33.786681 31838 consensus_queue.cc:260] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.788020 32022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.788763 32022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:33.789577 31838 sys_catalog.cc:564] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 11:12:33.789670 31813 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 11:12:33.789881 31813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:33.793535 31813 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.795872 31813 leader_election.cc:288] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [CANDIDATE]: Term 1 election: Requested vote from peers 8eae95a4384c438f867b15d305059776 (127.8.181.61:38125), e929cb01e48d4138a9d531f0796b4292 (127.8.181.60:38125)
W20260925 11:12:33.796011 31792 tablet.cc:2369] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:12:33.797255 31887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8eae95a4384c438f867b15d305059776"
I20260925 11:12:33.797776 31887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:33.799528 31962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e929cb01e48d4138a9d531f0796b4292"
I20260925 11:12:33.800031 31962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [term 0 FOLLOWER]: Advancing to term 1
I20260925 11:12:33.801347 31887 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79e2978d8844ce6ab2cb8cbcf6908cc in term 1.
I20260925 11:12:33.802048 31787 leader_election.cc:302] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eae95a4384c438f867b15d305059776, c79e2978d8844ce6ab2cb8cbcf6908cc; no voters:
I20260925 11:12:33.802675 32022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 11:12:33.804831 31962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79e2978d8844ce6ab2cb8cbcf6908cc in term 1.
I20260925 11:12:33.806924 32022 raft_consensus.cc:686] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [term 1 LEADER]: Becoming Leader. State: Replica: c79e2978d8844ce6ab2cb8cbcf6908cc, State: Running, Role: LEADER
I20260925 11:12:33.807556 32022 consensus_queue.cc:237] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } }
I20260925 11:12:33.816107 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79e2978d8844ce6ab2cb8cbcf6908cc. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.816694 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:33.822144 32029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 11:12:33.834569 32029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 11:12:33.857851 31866 tablet.cc:2369] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 11:12:33.877173 31962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [term 1 FOLLOWER]: Refusing update from remote peer c79e2978d8844ce6ab2cb8cbcf6908cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:12:33.880127 31887 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [term 1 FOLLOWER]: Refusing update from remote peer c79e2978d8844ce6ab2cb8cbcf6908cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 11:12:33.881683 32025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 11:12:33.882513 32025 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 11:12:33.893816 31908 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.8.181.61:38125
I20260925 11:12:33.943146 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.842727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.8.181.62:39329)
0925 11:12:33.842924 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.842930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.861558 (+ 18628us) server_negotiation.cc:407] Connection header received
0925 11:12:33.861875 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.861881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.861978 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.862060 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.870294 (+ 8234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.900564 (+ 30270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.922185 (+ 21621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.923020 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.923337 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.934815 (+ 11478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.934824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.934829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.934923 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.939166 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.940349 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.940355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.940359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.940530 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:33.941094 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:33.941100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:33.941104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:33.941500 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:33.941794 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:33.942446 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 11:12:33.942751 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 11:12:33.951388 31994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79e2978d8844ce6ab2cb8cbcf6908cc. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.951795 31994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:33.965306 32038 mvcc.cc:205] Tried to move back new op lower bound from 7333211151795486720 to 7333211151603150848. Current Snapshot: MvccSnapshot[applied={T|T < 7333211151795486720}]
I20260925 11:12:33.966001 32042 mvcc.cc:205] Tried to move back new op lower bound from 7333211151795486720 to 7333211151603150848. Current Snapshot: MvccSnapshot[applied={T|T < 7333211151795486720}]
I20260925 11:12:33.977469 32008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79e2978d8844ce6ab2cb8cbcf6908cc. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:33.977900 32008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:34.009370 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.860480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.8.181.61:38125)
0925 11:12:33.860709 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.860716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.872437 (+ 11721us) server_negotiation.cc:407] Connection header received
0925 11:12:33.911088 (+ 38651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.911096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.911203 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.911280 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.942630 (+ 31350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.943813 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.946157 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.947061 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.947203 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:33.994264 (+ 47061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:33.994275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:33.994281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:33.994340 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:33.998550 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:34.006058 (+ 7508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:34.006066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:34.006070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:34.006290 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:34.006905 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:34.006911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:34.006915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:34.007362 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:34.007671 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.008664 (+ 993us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.008974 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 11:12:34.011487 32043 mvcc.cc:205] Tried to move back new op lower bound from 7333211151795486720 to 7333211151603150848. Current Snapshot: MvccSnapshot[applied={T|T < 7333211151795486720}]
I20260925 11:12:34.024833 31994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:34.025194 31994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:34.031155 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:34.031490 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:34.032246 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:34.032510 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc [sys.catalog]: This master's current role is: LEADER
I20260925 11:12:34.042410 32008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79e2978d8844ce6ab2cb8cbcf6908cc" member_type: VOTER last_known_addr { host: "127.8.181.62" port: 39329 } } peers { permanent_uuid: "8eae95a4384c438f867b15d305059776" member_type: VOTER last_known_addr { host: "127.8.181.61" port: 38125 } } peers { permanent_uuid: "e929cb01e48d4138a9d531f0796b4292" member_type: VOTER last_known_addr { host: "127.8.181.60" port: 38125 } } }
I20260925 11:12:34.042728 32008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 11:12:34.045516 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:33.837002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.8.181.60:38125)
0925 11:12:33.837289 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:33.837299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:33.868500 (+ 31201us) server_negotiation.cc:407] Connection header received
0925 11:12:33.871352 (+ 2852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:33.871359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:33.871466 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:33.871549 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:33.916821 (+ 45272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.917956 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.947998 (+ 30042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:33.948795 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:33.948915 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.032984 (+ 84069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 11:12:34.032991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 11:12:34.032996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 11:12:34.033049 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 11:12:34.037408 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:34.038658 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:34.038666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:34.038670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:34.038828 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 11:12:34.039310 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 11:12:34.039315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 11:12:34.039319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 11:12:34.039770 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 11:12:34.040046 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.044784 (+ 4738us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.045077 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 11:12:34.047701 32029 catalog_manager.cc:1347] Generated new cluster ID: bca1b837e9b14e07800a4f1307051e15
I20260925 11:12:34.047813 32029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 11:12:34.085752 32029 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 11:12:34.087949 32029 catalog_manager.cc:1504] Loading token signing keys...
I20260925 11:12:34.122681 32029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c79e2978d8844ce6ab2cb8cbcf6908cc: Generated new TSK 0
I20260925 11:12:34.124305 32029 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 11:12:34.216159 31991 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.208966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.8.181.62:39329)
0925 11:12:34.209229 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.209238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.213329 (+ 4091us) server_negotiation.cc:407] Connection header received
0925 11:12:34.214381 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.214389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.214545 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.214633 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.215871 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41482: BlockingRecv error: recv got EOF from 127.0.0.1:41482 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 11:12:34.220513 31990 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.209807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.8.181.60:38125)
0925 11:12:34.210000 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.210007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.217916 (+ 7909us) server_negotiation.cc:407] Connection header received
0925 11:12:34.218874 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.218880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.218982 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.219062 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.220234 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40836: BlockingRecv error: recv got EOF from 127.0.0.1:40836 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 11:12:34.224334 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.213546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.8.181.61:38125)
0925 11:12:34.213752 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.213758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.221908 (+ 8150us) server_negotiation.cc:407] Connection header received
0925 11:12:34.222816 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.222823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.222918 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.223038 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.224069 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 11:12:34.230892 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.228445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.8.181.61:38125)
0925 11:12:34.228620 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.228626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.229127 (+ 501us) server_negotiation.cc:407] Connection header received
0925 11:12:34.229388 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.229394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.229532 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.229666 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.230586 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33372: BlockingRecv error: recv got EOF from 127.0.0.1:33372 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 11:12:34.236943 31911 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.8.181.61:38125: Invalid argument: socket address is ambiguous
W20260925 11:12:34.239122 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.237315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.8.181.61:38125)
0925 11:12:34.237487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.237495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.237517 (+ 22us) server_negotiation.cc:407] Connection header received
0925 11:12:34.237748 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.237755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.237844 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.237954 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.238878 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33386: BlockingRecv error: recv got EOF from 127.0.0.1:33386 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 11:12:34.247941 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.245415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.8.181.61:38125)
0925 11:12:34.245606 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.245612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.246161 (+ 549us) server_negotiation.cc:407] Connection header received
0925 11:12:34.246384 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.246389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.246529 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.246637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.247679 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33402: BlockingRecv error: recv got EOF from 127.0.0.1:33402 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 11:12:34.258003 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.255556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.8.181.61:38125)
0925 11:12:34.255802 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.255808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.256388 (+ 580us) server_negotiation.cc:407] Connection header received
0925 11:12:34.256598 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.256604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.256755 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.256884 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.257763 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33406: BlockingRecv error: recv got EOF from 127.0.0.1:33406 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260925 11:12:34.274583 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.271901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.8.181.61:38125)
0925 11:12:34.272110 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.272119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.272844 (+ 725us) server_negotiation.cc:407] Connection header received
0925 11:12:34.273084 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.273090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.273183 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.273296 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.274290 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33420: BlockingRecv error: recv got EOF from 127.0.0.1:33420 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 11:12:34.299077 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.296258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.8.181.61:38125)
0925 11:12:34.296453 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.296460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.297055 (+ 595us) server_negotiation.cc:407] Connection header received
0925 11:12:34.297289 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.297295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.297396 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.297546 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.298829 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33422: BlockingRecv error: recv got EOF from 127.0.0.1:33422 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 11:12:34.339077 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.336695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.8.181.61:38125)
0925 11:12:34.336945 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.336951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.337370 (+ 419us) server_negotiation.cc:407] Connection header received
0925 11:12:34.337605 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.337614 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.337732 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.337821 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.338816 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33428: BlockingRecv error: recv got EOF from 127.0.0.1:33428 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 11:12:34.409478 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.407141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.8.181.61:38125)
0925 11:12:34.407397 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.407403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.407973 (+ 570us) server_negotiation.cc:407] Connection header received
0925 11:12:34.408148 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.408152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.408230 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.408314 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.409237 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33444: BlockingRecv error: recv got EOF from 127.0.0.1:33444 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 11:12:34.477528 31983 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.8.181.60:38125
W20260925 11:12:34.542977 31917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 11:12:34.541147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.8.181.61:38125)
0925 11:12:34.541328 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.541332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.541648 (+ 316us) server_negotiation.cc:407] Connection header received
0925 11:12:34.541773 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.541777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.541853 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.541955 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 11:12:34.542783 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33460: BlockingRecv error: recv got EOF from 127.0.0.1:33460 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 11:12:34.571679 32005 catalog_manager.cc:1259] Loaded cluster ID: bca1b837e9b14e07800a4f1307051e15
I20260925 11:12:34.571786 32005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: loading cluster ID for follower catalog manager: success
I20260925 11:12:34.574528 32005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: acquiring CA information for follower catalog manager: success
I20260925 11:12:34.576581 32005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e929cb01e48d4138a9d531f0796b4292: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:12:34.687470 32019 catalog_manager.cc:1259] Loaded cluster ID: bca1b837e9b14e07800a4f1307051e15
I20260925 11:12:34.687644 32019 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: loading cluster ID for follower catalog manager: success
I20260925 11:12:34.690230 32019 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: acquiring CA information for follower catalog manager: success
I20260925 11:12:34.692190 32019 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8eae95a4384c438f867b15d305059776: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 11:12:34.809424 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:34.803566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.8.181.61:38125)
0925 11:12:34.803750 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.803754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.803897 (+ 143us) server_negotiation.cc:407] Connection header received
0925 11:12:34.804046 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.804050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.804276 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.804392 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:34.804843 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.805612 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.807210 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.807636 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.807744 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.808232 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:34.808807 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:34.808924 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.809122 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.809194 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 11:12:34.822613 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:34.816033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.8.181.60:38125)
0925 11:12:34.816515 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.816519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.816539 (+ 20us) server_negotiation.cc:407] Connection header received
0925 11:12:34.816746 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.816749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.816908 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.817069 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:34.817536 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.818323 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.819811 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.820281 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.820384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.820984 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:34.821759 (+ 775us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:34.821893 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.822139 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.822215 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":276,"threads_started":1}
W20260925 11:12:34.829578 31986 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.8.181.60:38125: Invalid argument: socket address is ambiguous
I20260925 11:12:34.838312 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:34.829996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.8.181.60:38125)
0925 11:12:34.830301 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.830310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.831846 (+ 1536us) server_negotiation.cc:407] Connection header received
0925 11:12:34.832138 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.832143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.832230 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.832295 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:34.833272 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.834447 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.836466 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.836892 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.837062 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.837531 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:34.837696 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:34.837787 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.837993 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.838085 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260925 11:12:34.839598 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:34.829607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.8.181.62:39329)
0925 11:12:34.831961 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.831967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.831985 (+ 18us) server_negotiation.cc:407] Connection header received
0925 11:12:34.832133 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.832140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.832400 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.832587 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:34.833646 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.834733 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.836844 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.837494 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.837634 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.838336 (+ 702us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:34.838906 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:34.839021 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.839134 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.839242 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2235,"thread_start_us":2138,"threads_started":1}
I20260925 11:12:34.840195 31917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 11:12:34.831188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.8.181.61:38125)
0925 11:12:34.831400 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 11:12:34.831408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 11:12:34.832417 (+ 1009us) server_negotiation.cc:407] Connection header received
0925 11:12:34.832633 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 11:12:34.832639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 11:12:34.832716 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 11:12:34.832827 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 11:12:34.834326 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.835613 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.837891 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 11:12:34.838549 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 11:12:34.838644 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 11:12:34.839392 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 11:12:34.839516 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 11:12:34.839583 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 11:12:34.839820 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 11:12:34.839914 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 11:12:34.845292 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31771
I20260925 11:12:34.846616 31833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:34.984573 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31771
I20260925 11:12:35.006628 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31845
I20260925 11:12:35.008239 31907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:35.123265 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31845
I20260925 11:12:35.145576 8916 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu pid 31920
I20260925 11:12:35.146916 31982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 11:12:35.258435 8916 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskA5RC6A/build/asan/bin/kudu with pid 31920
2026-09-25T11:12:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4839 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9969 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (378898 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST